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ERGODIC  CONTROL  OF  SWITCHING  DIFFUSIONS* 


Mrinal  K.  Ghosh]-,  Aristotle  ArapostathisJ  and  Steven  I.  Marcus§ 

January  30,  1996 

Abstract.  We  study  the  ergodic  control  problem  of  switching  diffusions  representing  a  typical 
hybrid  system  that  arises  in  numerous  applications  such  as  fault-tolerant  control  systems, 
flexible  manufacturing  systems,  etc.  Under  fairly  general  conditions,  we  establish  the  existence 
of  a  stable,  nonrandomized  Markov  policy  which  almost  surely  minimizes  the  pathwise  long-run 
average  cost.  We  then  study  the  corresponding  Hamilton-Jacobi-Bellman  (HJB)  equation  and 
establish  the  existence  of  a  unique  solution  in  a  certain  class.  Using  this,  we  characterize  the 
optimal  policy  as  a  minimizing  selector  of  the  Hamiltonian  associated  with  the  HJB  equations. 
As  an  example  we  apply  the  results  to  a  failure  prone  manufacturing  system  and  obtain  closed 
form  solutions  for  the  optimal  policy. 


1.  Introduction.  We  address  the  problem  of  controlling  switching  diffusions  by  contin¬ 
ually  monitoring  the  continuous  and  discrete  component  of  the  state.  The  objective  is  to 
minimize,  almost  surely,  the  pathwise  long-run  average  (ergodic)  cost  over  all  admissible 
policies.  A  controlled  switching  diffusion  is  a  typical  example  of  a  hybrid  system  which 
arises  in  numerous  applications  of  systems  with  multiple  modes  or  failure  modes,  such  as 
fault  tolerant  control  systems,  multiple  target  tracking,  flexible  manufacturing  systems  etc. 
[13],  [14],  [22].  The  state  of  the  system  at  time  t  is  given  by  a  pair  (X(t),S(t))  €  x  S, 
<S  =  {1,2,...,  N}.  The  continuous  component  X(t)  is  governed  by  a  “controlled  diffusion 
process”  with  a  drift  vector  which  depends  on  the  discrete  component  5(f).  Thus,  X(t) 
switches  from  one  diffusion  path  to  another  as  the  discrete  component  5(f)  jumps  from  one 
state  to  another.  On  the  other  hand,  the  discrete  component  5(f)  is  a  “controlled  Markov 
chain”  with  a  transition  matrix  depending  on  the  continuous  component.  The  evolution  of 
the  process  (X (f),  S(t)')  is  governed  by  the  following  equations: 

(1.1)  dX(t)  =b(X(t),S(t),u(t))dt  +  a(X(t),S(t))dW(t), 
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(1.2)  P(S{t  +  6t)=j  |  S{t)  =i,X{s),S{s),s<t )  =  A i:i(X(t),u(t))6t  +  o(6t),  t  /  j, 

for  t  >  0,  X(0)  =  X0,  5(0)  =  So,  where  6,  a,  A  are  suitable  functions,  A ij  >  0  for  i  ^  j , 
£f=i  Ajj  =  0,  W(-)  is  a  standard  Brownian  motion  and  u(-)  is  a  non-anticipative  control 
process  (admissible  policy).  The  latter  is  called  a  Markov  policy  if  u{t)  =  v(X(t),S(t)) 
for  a  suitable  function  v.  Our  goal  is  to  minimize  almost  surely  (a.s.)  over  all  admissible 
policies  the  functional 

1  fT 

(1.3)  limsup  — ;  /  c(X(t),  S(t),u(t))dt , 

T— >oo  J-  Jo 

where  c  is  the  running-cost  function.  Note  that  in  (1.3)  there  is  no  expectation;  we  are 
minimizing  the  limiting  pathwise  average  cost.  Such  a  criterion  is  very  important  in  practical 
applications  since  we  often  deal  with  a  single  realization.  Under  certain  conditions,  we  show 
that  there  exists  a  Markov  policy  v*  and  constant  p*  such  that 

1  fT 

lim  -  /  c(X{t),S{t),v*(X(t),S{t)))dt  =  p* ,  a.s. 

1  ~*oo  1  Jo 

and  for  any  other  admissible  policy  v(-) 

1  fT 

liminf—  /  c(X(t),  S(t),v(t))dt  >  p* ,  a.s. 

T-*oo  1  Jo 

This  establishes  that  v*  is  optimal  in  a  much  stronger  sense;  viz.,  the  most  “pessimistic” 
average  cost  under  v*  is  no  worse  than  the  most  “optimistic”  average  cost  under  any  other 
admissible  policy.  Also,  under  the  conditions  assumed  in  this  paper,  the  optimal  pathwise 
average  cost  coincides  with  the  optimal  expected  average  cost.  So  we  do  not  distinguish 
between  these  two  criteria. 

Our  paper  is  organized  as  follows.  In  Section  2  we  present  and  analyze  a  motivating  exam¬ 
ple,  while  in  Section  3  we  introduce  a  concise  mathematical  model  of  the  switching  diffusion. 
Section  4  is  devoted  to  the  study  of  recurrence  and  ergodicity  of  switching  diffusions.  The 
existence  of  an  optimal  policy  is  established  in  Section  5.  The  Hamilton- Jacobi-Bellman 
(HJB)  equations  are  studied  in  Section  6.  Conclusions  are  in  Section  7. 

2.  A  Motivating  Example.  The  failure  prone  manufacturing  system  presented  in  [1], 
[5],  [14],  is  a  very  good  example  of  the  class  of  systems  studied  in  this  paper.  This  section 
is  devoted  to  the  analysis  of  this  manufacturing  model.  Results  from  subsequent  sections 
will  be  used  in  this  example  and  thus  the  reader  will  have  the  opportunity  to  glimpse  at 
some  of  the  key  developments  of  the  paper. 

Suppose  there  is  one  machine  producing  a  single  commodity.  We  assume  that  the  demand 
rate  is  a  constant  d  >  0.  Let  the  machine  state  S(t )  take  values  in  {0, 1},  S(t)  =  0  or  1, 
according  as  the  machine  is  down  or  functional.  We  model  S(t)  as  a  continuous  time  Markov 
chain  with  generator 

—  Ao  A0 

.  Ai  -Aj  ’ 
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where  A0  and  Ai  are  positive  constants  corresponding  to  the  infinitesimal  rates  of  repair 
and  failure  respectively.  The  inventory  X  ( t )  is  governed  by  the  Ito  equation 


(2.1) 


dX ( t )  =  (u{t)  —  (i)  dt  +  adW (t) , 


where  a  >  0,  u(t)  is  the  production  rate  and  W(t)  is  a  one-dimensional  Wiener  process 
independent  of  S(t) .  The  last  term  in  (2.1)  can  be  interpreted  as  “sales  return",  “inven¬ 
tory  spoilage”,  “sudden  demand  fluctuations”,  etc.  A  negative  value  of  X{t)  represents 
backlogged  demand.  The  production  rate  is  constrained  by 


J  {0},  if  S(t)  =  0 
u(t)  €  < 

1  [0,r-],  if  S(t)  —  1. 

Let  c  :  R  — *  M+  be  the  cost  function  which  is  assumed  to  be  convex  and  Lipschitz.  Also 
c(^)  >  fl'(M)  for  some  increasing  function  g  :  M+  R+.  Thus,  c  satisfies  (5.3),  a  required 
condition  for  the  results  we  apply  to  hold.  We  show  later  in  this  section  that  a  certain 
hedging-point  policy  is  stable.  Therefore,  by  the  results  of  Section  5  there  exists  an  a.s. 
optimal  nonrandomized  Markov  policy  with  respect  to  the  cost  criterion 


i  rT 

lim  sup  —  /  c(X(t))dt. 

T— >oo  d  Jo 

The  HJB  equations  in  this  case  are 


3TV"(x,0)-dV'(x,0) 

(2.2)  I  2  |  + 

^V"(x,l)+  min  {(u  —  d)V'(x,  1)} 

«€[0,r]  ’ 


— Ao  Ao 
Ai  — Ai 


'^,0)\ 
V(x,l )) 

+  f  j  c(x)  = 


The  results  of  Section  6  ensure  existence  of  a  C 2  solution  (V,  p*)  of  (2.2),  where  p*  is  the 
optimal  cost.  Using  the  convexity  of  c(-),  it  can  be  shown  that  V(-,i)  is  convex  for  each  i. 
Hence,  there  exists  an  x*  such  that 

.  V'(x,  1)  <  0,  for  x  <  x* 

(2.3) 

V'(x,  1)  >  0,  for  x  >  x*. 


It  follows,  from  (2.3),  that  the  value  of  u  which  minimizes  (u  —  d)V'(x,  1)  is 

{r,  if  x  <  x* 

0,  if  x  >  x*. 

Since  V'(x*,  1)  =  0,  any  u  €  [0,  r]  minimizes  (u  -  d)V'(x*,  1).  Therefore,  in  view  of  The¬ 
orem  6.2,  the  action  u  e  [0,  r]  can  be  chosen  arbitrarily  at  x  =  x*.  To  be  specific,  we  let 
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u(x*)  =  d,  i.e.,  we  produce  at  the  level  that  meets  the  demand  exactly.  Thus,  the  following 
stable,  nonrandomized  Markov  policy  is  optimal 

{r,  if  x  <  x* 
d,  if  x  =  x* 

0,  if  x  >  x*. 


Note  that  the  stability  of  the  policy  (2.4)  follows  from  Theorem  6.3  provided  that  the  set  of 
stable,  nonrandomized  Markov  policies  is  nonempty.  We  show  next  that  the  zero- inventory 
policy  v  given  by 


(2.5) 


v(x,  0)  =  0,  v(x,  1) 


r,  if  x  <  0 
0,  if  x  >  0 


is  stable  if  and  only  if 

(2.6) 


(r~d)  ± 

Ai  A0' 


The  condition  (2.6)  is  in  accord  with  intuition.  Note  that  Aq  1  and  Af 1  are  the  mean 
sojourn  times  of  the  chain  in  states  0  and  1  respectively.  In  state  0  the  mean  inventory 
depletes  at  a  rate  d  while  in  state  1  it  builds  up  at  a  rate  ( r—d ).  Thus,  if  (2.6)  is  satisfied,  one 
would  expect  the  zero-inventory  policy  to  stabilize  the  system.  Our  analysis  confirms  this 
intuition.  We  first  show  that  under  v  the  process  (X(-),  5(-))  has  an  invariant  probability 
measure  rjv  with  a  strictly  positive  density.  In  view  of  Lemma  4.1,  it  then  follows  from  the 
ergodic  theory  of  Markov  processes  [24,  Chap.  1]  that  (X(-),  5(-))  is  positive  recurrent,  or 
equivalently  that  v  is  stable. 

By  Lemma  5.2,  the  density  <p  of  the  invariant  probability  measure  rjv  can  be  obtained  by 
solving  the  adjoint  system 


(2.7) 


(lw)V0M)  =  o, 


subject  to 

(2.8)  <p(x,i)>0,  ^2  /  <p(x,i)dx  =  1, 

*€{0,1} 

where  Lv  is  the  differential  generator  defined  in  (3.6)-(3.8).  Define 


-  2Aq  r  2Ai 

Aq  — y,  Ai  — — • 


-2  d  ,  _  2  r 

d  :=  — -  and  r  :=  — - 


cr* 


a* 


<7C 


Then  (2.7)  is  equivalent  to 


<p"{x,  0)  +  dtp'(x,  0)  —  A0c^>(rr,  0)  +  Ai<^(x,  1)  =  0 
(2.9a)  _  for  x  >  0, 

<p"(x,  1)  +  d<p'(x,  1)  -  Ai tp(x,  1)  +  A0 <p(x,  0)  =  0 


<p"(x,  0)  +  dtp'{x,  0)  —  Xo^ix,  0)  +  Ai  ip(x,  1)  =  0 
(2.9b)  _  _  _  for  x  <  0. 

1 )  —  (r  -  d)<p'(x,  1)  -  Ai <p(x,  1)  +  A0 y?(x,  0)  =  0 
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A  solution  of  (2.9),  subject  to  the  constraint  (2.8),  exists  if  and  only  if  (2.6)  holds  and 
takes  the  form: 


(2.10)  tp(x)  = 


'<p(x>  o)' 

1). 


ai  i  -1  )  e  Sl*  +  °2  (  Al  )  e  S2X,  for  x  >  0, 
.  Ao  )  \  Ai 


«3 


Ai 

0(®3) 


eS3X  +  04 


( ) 

V  b(*4)  / 


gS4X,  f0r  X  <  0, 


where  -0(s)  =  s2  +  Js  —  Ao,  Si  =  d,  S2  =  |  +  |  [d2  +  4(A0  +  Ai)]  V2,  and  S3,  S4  are  the  positive 
roots  of  the  polynomial 

s3  —  (r  —  2d)s2  —  [(r  —  d)d  +  Ao  +  Ai]s  +  [(f  —  d)A0  —  dA*)]  , 


ordered  by  0  <  S3  <  S4.  Also,  the  coefhcients  {04,02,03,04},  are  given  by: 


(2.11) 


1  f  («4  —  S3)S2  S4  +  S2  S3  +  S2  1 

0,1  —  —  1  — - - - - 1 - — - —  1 

^  '  Ao  +  Ai  s3  +  d  S4  +  d  * 

1  (s4  -  s3)s2 

02  =  -7 - = - ~ - 

A  Ao  +  Ai 

1  S4  +  S2 

a3__  AlT+d 

1  S3  +  S2 

a\  —  — - - 

^  54  +  d 

^  _  (S4  —  S3)(S2  ~  d)  Aq  +  Ai  f  S4  +  S2  _  £3_+_£2  1 
d  J  S  3  S4  / 


Note  that  if  ipx*  (•)  denotes  the  density  of  the  invariant  measure  corresponding  to  a  hedging- 
point  policy  as  in  (2.4),  then 

¥V(z)  =  y{x  ~  x*) . 

Given  a  convex  cost  function,  the  average  cost  p(x*)  corresponding  to  such  a  policy  can  be 
readily  computed  and  is  a  convex  function  of  the  threshold  value  x*. 

In  [5],  Bielecki  and  Kumar  have  studied  the  mean  square  stability  of  the  piecewise  de¬ 
terministic  system,  i.e.,  (2.1)  with  a  =  0.  They  have  shown  that  under  (2.6)  the  policy 
(2.5)  is  mean  square  stable,  and  have  computed  the  optimal  threshold  value  x*  in  (2.4). 
These  results  can  be  easily  reproduced  here  by  computing  the  limiting  value  of  the  invari¬ 
ant  distribution  as  o  — >  0,  which  we  do  next.  The  roots  S2,  s3  and  .sq  have  the  following 
asymptotic  dependence  on  a: 


(2.12)  s2  =  ^  +  e>(  1), 


(7 


s3  = 


(r  —  g?)Ao  —  dAi 
d(r  —  d) 


+  O(o2 


s4  = 


2  (r  -  d) 


+  0(1). 
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Thus,  using  (2.11),  we  obtain 


(2.13) 


0,2 


a  3 


04 


d[(r-rf)A0-dAi] 
r(  A0  +  Ax)2 


+  0(a2) 


a2  [(r  _  d)Xo  -  dAX]  4 
2  d(r  —  d)(Ao  +  Aj) 


cr2  d[(r  —  d)Ao  —  dAi] 
2  r'2(r  -  d)( A0  +  Ai) 


+  O(o4) . 


Let 


(r  —  d)  Ao  —  rfAj 
a°  d(r  -  d) 


and  6z(x)  denote  the  Dirac  measure  centered  at  z.  Using  (2.12)  and  (2.13),  we  can  show 
that  as  a  — >  0,  <px*{-)  converges  weakly  to  a  distribution  with  “density”  Tpx»  (•),  given  by 


wOM)  =  < 


/  Aiap 

Ao  +  A!e  > 

dan 
Aq+ A] 


<5i*  (x)  +  / 


dX\  qq 


(r—d)(  Ao+Ai ) 


oq0(x— a:*1 ) 


for  x  <  x*,  i  =  0, 
for  x  <  x*,  i  =  1, 


for  x  >  x*. 


Using  a  cost  as  in  [5]  of  the  form 


(2.14) 


c(x) 


c+  +  c 

2 


lx  + 
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with  c+  and  c  positive  constants,  the  average  cost  corresponding  to  the  policy  in  (2.4) 
takes  the  form: 


c(^)^x*  (x>  *) dx 


=  c+x* 


C+rAl  +  t\\{c+  +  c  ) 

(r  -  d)(A0  +  Ai)a0  (r  -  d)(A0  +  Ai)a0 


In  this  manner,  the  results  in  [5]  are  reproduced  exactly.  One  advantage  of  our  approach 
is  that  the  class  of  admissible  policies  does  not  have  to  be  restricted  as  is  done  in  [5],  in 
order  to  guarantee  the  existence  of  solutions.  With  our  method,  optimality  is  obtained  with 
respect  to  the  class  of  all  non-anticipative  policies.  Furthermore,  our  analysis  shows  that 
the  stability  of  the  zero-inventory  policy  is  retained  under  additive  noise  in  (2.1).  Let  us 
also  note  that  conditions  for  the  optimality  of  the  zero-inventory  policy  under  additive  noise 
can  be  readily  obtained  for  the  cost  in  (2.14)  using  the  density  in  (2.10). 

3.  The  Mathematical  Model.  We  first  exhibit  that  the  switching  diffusion  (1.1),  (1.2) 
can  be  constructed  on  a  given  probability  space.  Our  presentation  follows  [13],  [14];  we 
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repeat  it  here  for  the  sake  of  clarity  and  completeness.  Let  U  be  a  compact  metric  space, 
S  :=  {1,2,...,  N},  and 


b  =  [6l9 . . . ,  bd]'  :  Rd  x  <S  x  U  -»  Rd, 

G  —  [<7y(-,-)]  :  Rd  X  S  -»  Rdxd, 

A ij  :  Rd  x  U  — >  R,  i,j  G  5 , 

A,;7  >  0,  for  i  ^  j,  A ^  =  0,  for  any  i  €  S. 

j€S 


We  also  define  the  matrix  A  :  Rd  x  U  — >  RNxN  by 


[A(*.«)]y 


A  ij(x,u), 

0 ,  i  =  j. 


We  make  the  following  assumptions  which  will  be  in  effect  throughout  the  paper. 


Assumption  3.1. 

(i)  The  functions  b(x,k,u),  Gij(x,k )  and  \ij(x,u)  are  continuous  and  Lipschitz  in  x, 
uniformly  with  respect  to  u,  with  a  Lipschitz  constant  70.  Let  mo  denote  the  least 
upper  bound  0/  [|6(0,  fc,  -)||oo,  |tf7,(0,  k)\  and  ||Ay  (0,  -)|loo- 

(ii)  Gij(-,-)  is  uniformly  elliptic,  i.e.,  there  exists  a  constant  m  >  0  such  that 
er(-,  k)a'( •,  k )  >  ml. 

(iii)  The  matrix  A(x,  u)  is  irreducible  for  all  ( x ,  it)  £  X  U. 


For  a  Polish  space  Y,  <B(F)  denotes  its  Borel  n- field  and  V(Y)  the  space  of  probability 
measures  endowed  with  the  Prohorov  topology,  i.e.,  the  topology  of  weak  convergence.  Let 
9Jl(Y)  be  the  set  of  all  nonnegative,  integer-valued,  n-finite  measures  on  23(1”).  Let  0Jla(Y) 
be  the  smallest  c-field  on  9J l(Y)  with  respect  to  which  all  the  maps  from  9JI(Y)  to  N(J  {00} 
of  the  form  p  i-»  p(B)  with  B  €  23(Y)  are  measurable.  9Jl(Y)  is  assumed  to  be  endowed 
with  this  measurability  structure.  Let  V  =  V{U)  and  b  =  [&1; . . . ,  bd]  :  Rd  x  S  x  V  — >  Rd 
be  defined  by 

(3.1)  bi(-,-,v)=  f  bi(-,  •,  u)v(du) . 

Ju 

Similarly,  for  i,j€S  and  v  G  V,  is  defined  as 


(3.2) 


(•,  u)v(du) . 


For  i,j  G  <S,  x  €  and  v  G  V,  let  A ij(x,  v )  be  consecutive  (with  respect  to  the  lexicographic 
ordering  on  S  x  S),  left  closed,  right  open  intervals  of  the  real  line,  each  having  length 
A  ij(x,  v).  Define  a  function  h  :M.d  x  S  x  V  xR  — >  R  by 


(3.3) 


h(x,i,v,z ) 


j  —  i,  if  2  6  Aij  (x,  v) 
0,  otherwise. 
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Let  (X(t),  S(t))  be  the  (Rd  x  <S)-valued,  controlled,  switching  diffusion  process  given  by  the 
following  stochastic  differential  equations. 


dX(t)  =  b(X(t),S(t),  v(t))  dt  +  a(X{t),  S(t))dW(t) 

(S-4)  r 

dS(t)  =  /  h(X(t),  S(t— ),  w(t),  z)  p(dt,  dz) , 

Jr 

for  t  >  0  with  A"(0)  —  Xq,  5(0)  =  Sq,  where 

(i)  Xq  is  a  prescribed  Revalued  random  variable. 

(ii)  Sq  is  a  prescribed  <S-valued  random  variable. 

(iii)  W{-)=[W1(-),...,Wi(-)]>  is  a  d-dimensional  standard  Wiener  process. 

(iv)  p (dt,  dz)  is  an  9?t(R+  x  M)-valued  Poisson  random  measure  with  intensity  dt  x  m(dz), 
where  m  is  the  Lebesgue  measure  on  M. 

(v)  p(-,  •),  LW (•),  Xq  and  Sq  are  independent. 

(vi)  v(-)  is  a  V-valued  process  with  measurable  sample  paths  satisfying  the  non- 
anticipativity  property  that  the  cr-fields  St  and  3^^  given  by 

3t  =  <7{u(s),s  <  tj, 

S^  =  c7{W(S)-W(t),p(A,B)  :  A  €»([*, TO)),PeS3(R),5>t}. 
are  independent,  for  each  t  €  K. 

A  process  v(-)  satisfying  (vi)  is  called  an  admissible  (control)  policy.  If  v(-)  is  a  Dirac 
measure,  i.e.,  v(-)  =  6U^,  where  «(•)  is  17-valued,  then  it  is  called  an  admissible  nonran- 
domized  policy.  An  admissible  policy  is  called  feedback  if  v(-)  is  progressively  measurable 
with  respect  to  the  natural  filtration  St  =  {X(s),S(s),  s  <  t}. 

A  particular  subclass  of  feedback  policies  is  of  special  interest.  A  feedback  policy  v(-) 
is  called  a  (homogeneous)  Markov  policy  if  v(t)  =  v[X(t),S(t))  for  a  measurable  map 
v  :  x  S  — >  V.  With  an  abuse  in  notation  the  map  v  itself  is  called  a  Markov  policy. 

Let  II,  LLy/  and  II md  denote  the  sets  of  all  admissible,  Markov  and  nonrandomized  Markov 
policies  respectively. 

If  (W(-),P(-,  •)>  Xo,  Sq,v(-))  satisfying  (i)  -(vi)  above  are  given  on  a  prescribed  probability 
space  (f2,  0,  P),  then  under  Assumption  3.1,  (3.4)  admits  an  a.s.  unique  strong  solution  [17, 
Chap.  3],  and  X(-)  €  C(R+;Rd),  S(-)  €  D(R+;<S),  where  D(R+]S)  is  the  space  of  right 
continuous  functions  on  M+  with  left  limits  taking  values  in  S.  However,  if  ?.;(■)  is  a  feedback 
policy,  then  there  exists  a  measurable  map 

/  :  M+  x  <7(11+;  Rd)  x  D{ R+;  5)  —  V 

such  that  for  each  t  >  0,  v(t)  =  f(t,X(-),  S(-))  and  is  progressively  measurable  with  respect 
to  {5t}.  Thus,  v(-)  cannot  be  specified  a  priori  in  (3.4).  Instead,  one  has  to  replace  v(t)  by 
f  (t,  X (■) ,  S (■)) ,  and  (3.4)  takes  the  form 

dX(t)  =  b(X(t),S(t),  f(t,  X(-),  S(-)))dt  +  a(X(t),  S(t))dW(t ) 

(3-5)  f  , 

dS(t)  =  /  h{X(t),S(t-),f(t,X(-),S(-)),z)p(dt,dz), 

Js. 
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for  t  >  0  with  X (0)  =  X0 ,  S’(O)  =  So.  In  general,  (3.5)  does  not  even  admit  a  weak  solution. 
However,  if  the  feedback  policy  is  Markov,  then  the  existence  of  a  unique  strong  solution 
can  be  established. 

If  K(Rd)  is  a  vector  space  of  real  functions  over  Rd ,  we  adopt  the  notation  7C (]Rrf  x  S) 
to  indicate  the  space  (JC( Rd))  ,  endowed  with  the  product  topology.  For  example, 

I7(Md  x  S)  :=  {/  :  Rd  x  5  —  R  :  €  Lp(Rd),  for  all  i  €  5} 

and  similarly  we  define  Cfc(Rd  x  S),  Wk'p(Rd  x  5),  etc.  For  /  €  x  S)  and  u  &  U, 

we  write 

(3.6)  Luf(x,  k)  =  Llf( x,  k)  +  ^  A kj(x,  u)f(x,  j) , 

i&s 

where 

(3.7)  Lt  =  \H  a^x'  k)°Axi  k)dx.dx .  +  Wx'  k ’ u ^ 

i,j,e=  i  *  3  j= i  3 

and  more  generally,  for  v  G  V 

(3.8)  Lvf(x,  k)  =  [  Luf(x,  k)v(du) . 

Ju 

The  following  result  is  proved  in  [14]. 

Theorem  3.1.  Under  a  Markov  policy  v,  (3.4)  admits  an  a.s.  unique  strong  solution  such 
that  (X(-),S'(-))  is  a  strong  Feller  process  with  differential  generator  Lv . 

A  Markov  policy  v  is  called  stable  if  the  corresponding  process  (X(-),5(-))  is  positive 
recurrent.  In  this  case,  the  process  has  a  unique  invariant  probability  measure,  denoted  by 
rjv  €  V(Rd  x  5).  The  uniqueness  of  rjv  is  guaranteed  by  Assumption  3.1.  We  assume  that 
the  set  of  stable  Markov  policies  is  nonempty. 

The  Optimization  Problem.  Let  c  :  x  S  x  U  — >  M+  be  the  cost  function.  The 

following  assumption  on  the  cost,  c  will  be  in  effect  throughout  the  paper. 

Assumption  3.2.  For  each  i  E  S,  c(-,i ,  •)  is  continuous. 

We  define  c  :  x  S  x  V  — »  R+  by 

(3.9)  c(x,i,v)  =  /  c(x,i,u)v(du) . 

Ju 

Let  v(-)  be  an  admissible  policy  and  (X(-),S(-))  the  corresponding  process.  The  pathwise 
(long-run)  average  cost  incurred  under  v(-)  is 

T 

c(X(t),  S(t),  v(t))dt . 


(3.10) 


lim  sup  —  [ 
T  — >oo  T  J0 
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We  wish  to  a.s.  minimize  (3.10)  over  all  admissible  policies.  Our  goal  is  to  establish  the 
existence  of  a  stable  Markov  policy  which  is  a.s.  optimal.  In  general,  this  is  not  the  case,  as 
the  following  simple  counterexample  shows  [6].  Let  c(x,i )  =  exp(— ||x||2).  Then  for  every 
stable  Markov  policy  the  average  cost  is  positive  a.s.,  while  we  can  find  an  unstable  Markov 
policy  for  which  the  average  cost  is  a.s.  zero,  making  it  an  optimal  policy.  We  want  to 
rule  out  this  possibility,  as  stability  is  a  very  desirable  property.  We  carry  out  our  study 
under  two  alternate  sets  of  hypotheses:  (a)  a  condition  on  the  cost  which  penalizes  unstable 
behavior,  (b)  a  blanket  stability  condition  which  implies  that  all  Markov  policies  are  stable. 
We  describe  these  conditions  in  Section  6. 


4.  Recurrence,  Ergodicity  and  Harmonic  Functions  of  Switching  Diffusions.  Due 

to  the  interaction  between  the  continuous  and  discrete  components,  the  study  of  recurrence 
and  ergodicity  of  switching  diffusions  is  quite  involved.  Let  v  be  a  Markov  policy  which  will 
be  fixed  throughout  this  section  unless  explicitly  stated  otherwise.  Let  Pv  :  R+  x  Rd  x  S  — > 
■p(Rd  X  S)  denote  the  transition  function  of  the  corresponding  process  (X(-),  S'(-)).  Also 
i  and  Ef.  {  denote  the  probability  measure  and  the  expectation  operator,  respectively,  on 
the  canonical  space,  of  the  process  (Al(-),  S(-))  starting  at  (x,  i)  gR^XiS.  The  following 
result  plays  a  crucial  role  in  recurrence. 

Lemma  4.1.  For  any  ( t,x,i )  €  R+  x  Rd  x  S,  the  support  of  Pv(t,x,i]  •)  is  Rd  x  S. 


Proof.  For  each  i  G  <S,  let  Ti  denote  the  sojourn  time  of  S{t)  in  state  i.  Then 


PUn  >t)=K 


v 

x,i 


exp 


A^(X(S),u(X(s),5(S)))^ 


Let  A V.(5)  :=  Ay(X(s),u(X(s),5(s))),  IAj(s)  :=  I{X(s )  G  A,S{s)  =  j},  and  Pf  be  the 
transition  function  of  the  diffusion  corresponding  to  L j1,  i.e. ,  the  diffusion  with  no  switching 
and  S(t )  =  i.  For  A  €  ©(R^),  i,j£S  and  t  >  0, 


(4.1) 


=  Ki 


PEI 


Pv A  x  {j})  =  E*Xti[lAJ(t)  |  ^  >  t]P%ti(Tt  >  t)+Eli[lAd{t)I{ri  <  *}] 

Pf(t,x,A)6ij 

“,i[f  -AL(s)exp^  A L(s')ds')ds 

[  Pf(s,x,dy)^2xvik{s)Pv(t-  s,y,k,A  x  { j }) 
J&d  fc/i 

K.i  exp(  /  ASM*)  PHt,x,A)6„ 


+  E/‘ES, 

k^i 


-\,i(5)A-fc(s)exp  /  A  Ks^ds 


[  Pf  (s,  x,  dy)Pv  (t  —  s,y,  k,  A  x  {j})ds. 
J  Rd 
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Define  the  transition  matrix  IP  by 

[rP(t,x,  A)]ij  =  Pv(t,x,i,A  x  {j})  . 

Then  we  can  suitably  define  the  matrix  measures 

:RxR^  ( T(Rd))NxN 

with  Fj(t,  x,A)  positive,  diagonal  and  T^f,  x,  A)  non-negative,  irreducible  (by  Assump¬ 
tion  3.1  (iii)),  for  all  (t,x,A)  €  R+  x  x  provided  A  has  positive  Lebesgue 

measure,  so  as  to  write  (4.1)  in  the  form: 

(4.2)  n v(t,x,A)=Tv1(t,x,A)+  [  [  Tv2(s,x,dy)Uv(t-s,y,A)ds. 

Jo  Jmd 

The  desired  result  follows  from  (4.2),  using  the  irreducibility  of  T^t,  x,  A).  □ 

Let  th,  Tj  be  the  stopping  times  defined  as  follows: 

(4.3)  m  =  inf {i  >  0  :  S(t)  =  i  and  S(t')  ^  i,  for  some  0  <  t!  <  t) 

(4.4)  Tj  =  inf{t  >  0  :  S(t)  =  j}. 

Let  D  c  Rd  be  a  bounded  open  set  and  J  a  subset  of  S.  Define 

(4.5)  td,j  =  inf {i  >  0  :  (. X(t),S(t ))  0  D  x  J) 

(4.6)  tq  =  inf (t  >  0  :  X(t)  D}. 

Using  (4.2)  and  well  known  arguments  in  Markov  processes  [12,  Vol.  I,  p.  Ill]  the  following 
results  can  be  proved. 

Lemma  4.2.  If  t  is  a  stopping  time  of  the  form  Ttl,  Tj,  td,j  or  to,  as  defined  in  (4-3)- 
(4-6),  then,  for  each  compact  set  K  C  Md, 

sup  Evx  i[T\<  oo. 
i>enM,  xeK 

It  is  well  known  that  harmonic  functions  play  an  important  role  in  the  study  of  recurrence 
and  ergodicity  of  Markov  processes  [3].  Therefore,  we  now  turn  to  the  analysis  of  some 
properties  of  the  harmonic  functions  of  the  process  (X(-),  S(-))  under  the  Markov  policy  v. 
The  function  /  is  called  Lv -harmonic  in  D  if  it  is  bounded  on  compact  subsets  of  D,  and 
for  all  x  E  D,  i  €  S , 

(4.7)  f(x,i)  =  EvXtif{X(TV,j),S(TV,j)), 

for  every  neighborhood  V  of  x  having  compact  closure  V  in  D  and  every  subset  J  C  S 
containing  i.  It  is  clear  that  if  /  is  IA-harmonic  then 

(4.8)  f(x,i)  =  EU(X{tv),S{tv)). 

On  the  other  hand,  if  (4.8)  holds  then  by  conditioning  on  $Tv  j  we  obtain 
f{x,i)  =  Eltl[E-[f{X{TV),S{TV))  | 

=  Ki  ~  Tv ^  S{-Tv  ~  Tv^\ 

=  £%ti[f{X(TvtJ),S(iv,j))], 
concluding  that  (4.7)  and  (4.8)  are  actually  equivalent. 
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Lemma  4.3.  Let  D  C  Md  be  open.  Then, 

(i)  Every  Lv -harmonic  function  in  D  is  continuous  m  D. 

(ii)  If  Lv  f  =  0  in  D  and  f  G  W‘2,P(D  x  S),  then  f  is  Lv -harmonic.  Conversely,  if  f  is 
Lv -harmonic  and  f  G  W^{D  x  S),  then  Lv  f  =  0  in  D. 

(iii)  (Maximum  Principle)  Let  D  be  connected  and  f  >  0  and  Lv  -harmonic  in  D .  Then 
f  is  either  strictly  positive  in  D  x  S  or  identically  zero. 

Proof.  The  proof  of  (i)  is  standard  [3],  [12,  Vol.  II,  Chap.  13]  and  (ii)  can  easily  be  proved 
using  Ito’s  generalized  formula.  Let  xq  G  D,  io  G  S  and  r  >  0  be  such  that  f(xo,io)  =  0 
and  B(xq,  r)  C  D,  where  B(xq,  r)  =  {x  G  Rd  :  ||x  —  xo||  <  r }.  Then 

0  =  /(®o,*o)  =  f(yJ)Pxono(X(TB(xo,r))  €  dy,  S{rs(Xo,r) )  =j). 

jeSJdB(x0,r) 

Then,  by  Lemma  4.1,  we  can  show  using  standard  arguments  [16,  Chap.  6]  that  the  support 
of  the  measure  PWQ  {X[rB{xo^r))  G  dy,S{rB{xa,r ))  =  j)  is  <9 B(x0,r)  x  S.  Hence, 

f(y,j )  =  0,  for  all  y  G  dB(x0,r),  j  G  <S  . 

It  follows  that  the  set  {y  :  f(y,j)  =  0,  j  G  5}  is  open  in  D  and  since  D  is  connected  the 
result  follows.  □ 

We  next  state  Harnack’s  inequality  for  lA-harmonic  functions,  which  extends  a  very 
important  result  in  partial  differential  equations.  This  inequality  plays  a  crucial  role  in 
proving  the  existence  of  a  solution  to  the  HJB  equation  via  the  vanishing  discount  method, 
as  is  done  in  Section  6.  As  far  as  we  know,  this  result  is  not  known  in  the  literature  on 
partial  differential  equations.  The  detailed  proof  of  Harnack’s  inequality  is  quite  elaborate 
and  can  be  found  in  the  Appendix.  The  proof  follows  the  method  introduced  for  diffusions 
by  Krylov  and  Safonov  [18],  for  deriving  estimates  for  the  oscillation  of  a  harmonic  function. 
For  the  system  of  coupled  elliptic  operators  characterizing  switching  diffusions  considerable 
complications  arise  in  trying  to  follow  the  same  methodology,  due  to  the  vector-valued 
nature  of  the  IA-harmonic  functions.  A  crucial  step  in  the  proof  is  ‘coupling’  together  the 
oscillations  of  the  distinct  components  of  the  harmonic  function.  The  irreducibility  of  the 
matrix  A  is  essential  in  accomplishing  this  task. 

Theorem  4.1  (Harnack’s  Inequality).  Let  12  c  be  a  bounded  domain  and  K  C  12  a 
closed  set.  There  exists  a  constant  C  >  0,  depending  only  on  12,  K ,  the  dimension  d,  N, 
the  bounds  m,  mo  and  the  Lipschitz  constant  70  introduced  in  Assumption  3.1,  such  that  for 
any  nonnegative  function  f  G  W^f(f2  x  S),  p  G  [1,  00),  satisfying  Lv  f  =  0  in  12  x  S,  for 
some  Markov  policy  v, 

f(x,i)  <Cf{y,j) ,  Wx,yeK,  Vi,j£S. 

We  now  discuss  the  recurrence  properties  of  switching  diffusions.  Our  treatment  closely 
follows  [3],  therefore  we  skip  the  details  in  several  places.  A  point  (x,  i)  G  M.d  x  S  is  said  to 
be  recurrent  if  given  any  e  >  0, 

(4.16)  Pli{X{tn)  G  B{x,e),  S(tn )  =  i,  for  a  sequence  tn  |  00)  =  1. 
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A  point  (x,  i )  is  transient  if 


(4.17)  ^,i(IW)ll  -*  oo,  as  t  -»  oo)  =  1. 

If  all  points  of  the  switching  diffusion  are  recurrent,  then  it  is  called  recurrent.  A  transient 
switching  diffusion  is  similarly  defined.  Note  that  the  discrete  component  of  the  process 
has  been  ignored  in  the  definition  (4.17).  The  reason  for  doing  so  is  that,  in  view  of 
Assumption  3.1  (iii),  we  can  show  that  provided  that  the  continuous  component  visits  a 
bounded  set  infinitely  often  with  probability  1,  then  the  discrete  component  is  recurrent. 
More  generally,  a  switching  diffusion  exhibits  a  dichotomy  in  that  it  is  either  recurrent  or 
transient  as  we  will  later  show. 

Lemma  4.4.  The  following  statements  are  equivalent. 

(i)  The  switching  diffusion  is  recurrent; 

(ii)  Pf  i(X(t)  G  D,  S(t)  =  j,  for  some  t  >  0)  =  1,  for  any  open  set  D  C  M.d  and  any 
J  €  5. 

Proof.  We  prove  (i)  — >  (ii)  (the  converse  is  easier).  We  distinguish  two  cases: 

Case  1.  Let  x  G  D,  i  ^  j.  Let  B  =  B(x,e)  and  B,  be  bounded  open  sets  such  that  B  C  B i 
and  B\  C  D.  Let 

r/i  =  inf  {i  >  0  :  X(t)  G  <92?i}, 
and  inductively,  for  n  =  1, 2, . . . , 

r?2„  =  inf {t  >  rj2n-i  ■  X(t)  G  dB } 
r/2n+i  =  inf {t  >  r/2n  ■  X{t)  G  dBx). 

Then,  by  recurrence,  rjn  <  oo,  Pf  t  -  a.s.  Note  that 

pvAT(Bx{j}r  <rBi) 

is  L^-harmonic  in  B\  x  <S  and  not  identically  zero.  Therefore  by  Lemma  4.3, 

(4'18)  ,  .“L  PyAT(Bx{j}r  <rBl)>S i  >  0, 

(y,e)€BxS 

for  some  8\  >  0.  Next  we  define 

Ao  —  { S(t )  =  j,  for  some  t  G  [0 , 771 ) } 

An  =  { S(t )  =  j,  for  some  t  G  [7/2„,7?2n+i)}- 

By  (4.18)  and  the  strong  Markov  property, 

Pfi(At)  <  (1  -  8,1  P^(n  Ak)  ^  (!  -  «i)n+1- 

k= 0 
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Now, 


P'JA.Xit)  G  D,  S(t)  =  j,  for  no  t  >  0) 

<  g  BU  S(t)  =  J,  for  no  t  >  0) 

fc= 0 

Case  Suppose  x  ^  D  and  let  B  =  B(x,e),  B\  and  D\  be  bounded  open  sets  such  that 
B  p|  D  =  0,  B\  C  D  and  B  (J  Bi  C  D\.  Let 

rh  =  TDi 

%n  =  {*  >  »?2n-l  :  €  8B} 

V2n+1  =  {*  >  :  -Y(0  €  0Di}- 


Let  <?2  >  0  be  such  that 


<  T(Bx{i}Y 


)  >  s2  >  0. 


Define 

A'n  =  {X(t)  €  Bi,S(t )  =  j,  for  some  t  G  [ry2n-i, ??2n)}- 
Then,  as  in  the  previous  case, 

PT  (X(f)  €  L>,  5(f)  =  j,  for  no  t  >  0)  =  0.  □ 

In  view  of  Lemma  4.4,  the  following  results  can  be  proved  the  same  way  as  in  [3],  [4]. 

Lemma  4.5.  The  following  statements  are  equivalent. 

(i)  The  switching  diffusion  is  recurrent. 

(ii)  Pf  i (X (t)  e  D,  for  some  t  >  0)  =  1,  for  all  x  G  i  G  S  and  any  non-empty  open 
set  D. 

(iii)  There  exists  a  compact  set  K  C  Md  such  that  t  (X(t)  G  K,  for  some  t  >  0)  =  1 
for  all  (x,i)  G  Kd  x  S. 

(iv)  Pf  i(X(tn)  G  D,  for  a  sequence  tn  f  00)  =  1,  for  all  x  G  Md,  j  G  5  and  any 
non-empty  open  set  D . 

(v)  There  exists  a  point  z  G  Rd,  a  pair  of  numbers  ro,  r i,  0  <  r0  <  n,  and  a  point 
y  G  dB(z,r  1)  such  that  Pf^  (r^  ro^c  <  00)  =  1,  for  any  i  G  S. 

Theorem  4.2.  For  any  Markov  policy,  the  switching  diffusion  is  either  recurrent  or  tran¬ 
sient. 

A  recurrent  switching  diffusion  admits  a  unique  (up  to  a  constant  multiple)  cr-finite 
invariant  measure.  The  switching  diffusion  is  called  positive  recurrent  if  it  is  recurrent  and 
admits  a  finite  invariant  measure. 
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A  Markov  policy  v  is  called  stable  if  the  corresponding  process  is  positive  recurrent;  the 
corresponding  invariant  probability  measure  is  denoted  by  T]v. 

As  is  well  known  from  the  general  theory  of  dynamical  systems,  even  if  L V  generates  a 
positive  recurrent  diffusion,  for  each  i  €  S,  and  the  parametric  Markov  chain  is  ergodic, 
there  is  no  reason  to  expect  that  the  policy  v  is  stable,  i.e.,  the  switching  diffusion  is  positive 
recurrent.  Indeed,  as  the  following  example  shows,  the  hybrid  process  can  be  anything  from 
transient  to  positive  recurrent. 

Example  4-1 ■  We  first  consider  a  piecewise  deterministic  system  with  state  dependent  Mar¬ 
kovian  switching.  Let  E+,E-  C  R2  be  defined  as  follows: 


E+  =  {(^i,  £2)  :  xx  >  0}  lj{m2  <  0,xi  =  0} 
E-  =  {(xx,x2)  :  xx  <  0}  (J{x2  >  0,xi  =  0}. 

Let 

Consider  two  stable  dynamical  systems  V0  and  T>\  defined  by 


and 


V0  :  x  = 


V 1  :  x  = 


A0x, 

x  €  E+ 

A\x , 

x  e  E- 

Axx, 

x  e  E+ 

A0x, 

x  E  E~ 

For  8  >  0,  let  Z  be  a  (parameterized)  Markov  chain  taking  values  in  {0, 1}  with  rate  matrix 


8 

8  ' 

-  _  1 

1  - 

1 

L  s 

1 

s  -1 

on  E+ ,  and 

6 

.  8 

<5 

—8_ 

and  consider  the  dynamical  system 

v-.=  vz. 


If  we  define  77  by 


V  = 


Z ,  x €  E+ 

1  —  Z,  x  G  E _  , 


then  77  is  Markovian  with  rate  matrix 


—8  8  ' 

I  _i  > 

L  S  s  -1 


and  T>  can  represented  as 

x  =  Avx. 
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Define 


To(t)  =  {t  <t  :  tj{t)  =  0} 
Ti(t)  =  {t  <  t  :  ry(r)  =  1}, 


and  Ao (t)  =  m(T0(t)),  \i(t)  =  m(Ti(t)),  where  m  is  the  Lebesgue  measure  on  K+.  Then, 
the  solution  to  V  can  be  expressed  as 


x(t)  =  exp(2Ao(t)  —  3Ai(t)) 


cos  t 
sinf 


—  sint 
cos  t 


*(0). 


By  the  ergodic  theory  of  Markov  processes  [24,  Chap.  1],  as  t  — >  oo, 

82t  t 

Ao(<)~T7^' 

Thus, 

2  —  3S2 

2Ao(i)-3A1(«)~y-^ri. 

Therefore,  V  is  stable  for  6  <  and  unstable  for  6>yJ f.  The  matrices  Ao,  Ai  can  be 
suitably  altered  to  exhibit  various  other  possibilities. 

Now  let  X(t)  be  defined  as  dX(t)  =  An^X{t)dt  +  adW(t),  where  W{-)  is  a  standard  2- 
dimensional  Wiener  process  and  era'  is  a  2  x  2  positive  definite  matrix  with  constant  entries. 
Then  it  is  easily  shown  that  the  stability  (instability)  of  V  implies  the  positive  recurrence 
(transience)  of  X(t).  Note  that  in  this  example  the  drift  is  unbounded.  However,  in  the 
study  of  recurrence,  boundedness  of  the  drift  can  be  replaced  by  local  boundedness. 

Remark  4-1.  In  view  of  the  above  example,  it  is  clear  that  two  positive  recurrent  processes 
with  suitable  switching  may  result  in  a  transient  process.  Similarly,  the  random  combination 
of  two  transient  processes  may  give  rise  to  a  positive  recurrent  process.  This  phenomenon 
can  be  exploited  in  many  practical  situations  such  as  fault-tolerant  control  systems,  flexible 
manufacturing  systems,  etc.  In  a  control  system  with  multiple  modes,  we  can  trade  off 
the  stability  of  some  (or  all)  nodes  to  gain  a  desired  degree  of  flexibility.  Addition  of  a 
few  redundant  nodes  and/or  the  incorporation  of  a  suitable  switching  mechanism  among 
the  nodes  could  result  in  global  stability  of  the  system,  thereby  gaining  flexibility  without 
sacrificing  reliability. 

A  general  criterion  for  positive  recurrence  of  a  switching  diffusion  is  provided  by  the 
following  theorem: 

Theorem  4.3.  Let  z,  ro,  r\  be  as  in  Lemma  4-5(v).  Then  the  switching  diffusion  is  positive 
recurrent  if 


(4.19) 


sup 

yedB(z,n),iSS 


B(z.r0)c  J 


<  OO  . 


16 


The  proof  is  standard  [3].  Note  that  it  may  be  very  difficult  to  verify  (4.19)  for  general 
b,  a,  A.  One  usually  verifies  (4.19)  by  constructing  a  Lyapunov  function  [3].  For  switching 
diffusions  such  a  construction  seems  difficult,  since  it  involves  solving  a  system  of  ordi¬ 
nary  differential  equations  in  closed  form.  However,  we  present  some  criteria  for  positive 
recurrence  and  discuss  some  implications. 

(Cl)  There  exists  a  w  £  C2(Rd  x  S),  w  >  0  such  that 

(i)  w(x,i)  —*  oo,  as  ||x||  — >  oo. 

(ii)  For  each  v  €  11  m ,  [w(X(t),  5(f))]  and  E%ti\Lvw(X(t),  S(t))  |  are  locally 

bounded. 

(iii)  There  exists  p  >  0,  q  >  0  such  that  Luw(x,  i)  <  p  —  qw(x,  i ),  for  each  u  £  U. 

(C2)  There  exists  a  C2  function  w  :  x  <S  — »  M+  such  that 

(i)  limy-cH^oo  w(x,  i)  =  +oo. 

(ii)  There  exists  a  >  0  and  £  >  0  such  that  for  |(a:||  >  a,  Luw(x,  i )  <  —e,  for  all  u  €  U, 
i  €  S,  and  ||Vtc(x,  i)||2  >  m~1,  where  m  is  the  constant  in  Assumption  3.1  (ii). 

(iii)  w(x,i)  and  ||V'ia(x,  i)|j  have  polynomial  growth. 

Theorem  4.4.  Under  either  (Cl)  or  (C2),  the  process  under  any  Markov 

policy  v  is  positive  recurrent.  Thus,  all  Markov  policies  are  stable. 

Proof.  Under  (Cl),  the  result  follows  from  [24,  Theorem  25,  p.  70].  Under  (C2),  the  tech¬ 
nique  of  the  proof  of  [6,  Lemma  6.2.2,  p.  150]  can  be  closely  paralleled  to  draw  the  desired 
conclusion.  □ 

Remark  f.2.  If  a  —  I  and  b  is  such  that  (b(x,i,u),x)  <  ~{d  +  l)/2  for  all  i  €  S  and 
||x||  sufficiently  large,  then  w(x)  —  ||x||2  is  a  Lyapunov  function  for  the  system.  We  can 
construct  several  examples  using  this  idea.  Note  that  in  this  case  all  the  diffusion  generators 
Lf  give  rise  to  positive  recurrent  diffusions  and  have  a  common  Lyapunov  function,  (i.e., 
one  which  is  independent  of  i).  If  all  Lf  have  a  common  Lyapunov  function,  then  switching 
does  not  destabilize  the  hybrid  system.  Of  course,  this  is  a  very  strong  condition  and  is 
rarely  met. 

5.  Existence  of  an  Optimal  Policy.  In  this  section  we  establish  the  existence  of  a 
stable,  nonrandomized  Markov  optimal  policy  under  certain  conditions.  We  follow  the 
methodology  developed  in  [6],  [8],  [9],  [10]  for  controlled  diffusions.  For  switching  diffusions, 
similar  techniques  carry  through  with  some  extra  effort.  Therefore,  we  present  the  main 
ideas  skipping  some  of  the  technical  details. 

Let  n sm  and  TIsmd  denote  the  set  of  stable  Markov  and  stable  nonrandomized  Markov 
policies  respectively.  Since  we  are  searching  for  an  optimal  policy  in  n smd,  it  is  natural  to 
assume  that  H^w  is  nonempty.  Let  v  <E  n^M-  Then 

(5.1)  pv:=  W  c(x,i,v(x,i))r}v(dx,i) 

ies 

=  lim  ^  f  c(X(s),S(s),v{X(s),S(s)))ds ,  a.s. 

1  -too  1  J  Q 
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Let 

(5.2) 


P*  ~  {pv}- 

^srisA,/ 1 

We  assume  that  p*  <  oo.  We  now  state  a  condition  on  the  cost  function  which  penalizes 
unstable  behavior. 

(C3)  Assume  that  for  each  i  G  S , 

(5.3)  lim  inf  {  inf  c(x,  i,  u) }  >  p* . 

|| x (I  — >oo  uGu 

Intuitively,  (5.3)  penalizes  trajectories  lying  outside  the  set  inf  {c(x,i,u)\  <  p* ,  forcing 
an  optimal  process  to  spend  a  non-vanishing  fraction  of  time  in  a  bounded  neighborhood 
of  this  compact  set.  This  behavior  results  in  the  stability  of  every  optimal  policy.  If 
c(x,i,u )  =  iL(||x||)  for  some  increasing  function  K  :  R+  — *  R+  then  it  can  be  easily  seen 
that  (5.3)  holds.  Such  cost  functions  arise  quite  often  in  practice.  Condition  (C3)  is  referred 
to  as  the  near-monotonicity  condition  [6,  Chap.  6]. 

For  v  G  II <j m  (or  Usmd),  we  define  the  ergodic  occupation  measure  p[v]  G  ”P(Rd  x  S  x  17) 
as 

(5.4)  p[v\(dx,i,du)  =  rjv(dx,  i)v(x,  i)(du). 

Let 

h  =  [p[v]  ■  v  G  n5M} 

h  =  {p[V\  '■  v  €  IIsA/jr)}. 

The  following  results  can  be  proved  as  in  [10],  [14]. 

Lemma  5.1.  The  sets  I\,l2  are  closed,  I\  is  convex  and  the  set  of  extreme  points  of  I\ 
lies  in  1 2. 

Let  v(-)  be  an  arbitrary  admissible  policy.  Define  the  ^(R^  x5x  [7)-valued  empirical 
process  Pt(v),  for  t  >  0  by 

1 

(5.5)  pt{v)[A  x  {i}  x  B)  —  -  l{X(s)  G  A,  S(s)  —  ?}  v(s)(B)  ds  , 

t  Jo 

with  A  G  *B(Rrf),  B  G  !B(f7)  and  i  G  <5.  Let  Rd  =  Rd  (J  {00}  be  the  one-point  compactiiica- 
tion  of  Rd.  We  identify  pt(v)  with  an  element  of  "P(Rd  x  S  x  U)  by  assigning  zero  mass  at 
{00}  x  S  xU.  Since  T^R0*  x  S  x  U)  is  compact,  {pt{v)},  viewed  as  a  7?(  R  x  S  x  C/)-valued 
process,  converges  to  a  sample  path  dependent  compact  limit  set  in  "P(R  x  S  x  U).  Note 
that  any  element  p  £  V( R  x  S  x  U)  can  be  decomposed  as 

(5.6)  p(C)  -  h,,  p'(C  D  (Rd  xSxU))  +  (  1  -  <5m)m/'(C'D  ({00}  x  S  x  U)) , 

for  C  G  53  (Rd  x  S  x  17).  In  this  decomposition  6^  G  [0, 1]  is  always  uniquely  defined,  and 
pi  G  ^(R^  x  <S  x  U)  (respectively  p"  G  ^({oo}  x  S  x  U))  is  also  unique  if  8^  >  0  (resp. 
Sfj,  <  1).  We  may  render  pi ,  p"  unique  at  all  times  by  imposing  an  arbitrary  fixed  choice 
thereof  when  8 ^  =  0,  respectively,  1. 

Combining  the  results  in  [19]  with  the  technique  in  [6,  Lemma  6.1.1,  p.  144],  we  establish 
the  following  lemma. 


18 


Lemma  5.2.  If  p  E  "P(Rd  x  S )  satisfies 


(5.7)  V  [  Lvf(x ,  0  =  0,  v  /  €  H, 

ies  *'R<1 

for  some  Markov  policy  v,  where  H  is  a  dense  subset  of  Co(Rd  x  <S),  tfren  p  =  rjv . 

Proof.  Using  the  usual  approximation  procedure  we  can  show  that  (5.7)  is  true  for  all 
/  E  Cj*( Rd  x  5).  Let  (X(-),  5(-))  be  the  process  corresponding  to  the  policy  v  with  initial 
law  fi.  The  law  pt  of  this  process,  for  t  >  0,  satisfies  the  Kolmogorov  forward  equation 

y  /  f(x,i)pt(dx,i)  =  y  /  f(x,i)p(dx,i) +  Y]  /  Lv  f(x,i)ps(dx,i)  ds , 

igS  ./O 

for  all  /  £  (Rd  X  5).  The  uniqueness  of  the  solution  to  the  above  equation  is  established 
in  [19].  Since  pt  =  p  is  a  solution  to  (5.7),  it  follows  that  fi  =  tjv.  □ 

We  disintegrate  fi'  E  V(Rd  x  S  x  U)  as 


(5.8) 


p'(dx,i,du)  =  p*(dx,i)vfJ/(x,i)(du) , 


where  fi*  is  the  marginal  of  fi'  on  x  S  and  v ^  is  a  version  of  the  regular  conditional  law 
defined  fi*  -  a.s.  We  select  an  arbitrary  version  and  keep  it  fixed  henceforth.  Using  the 
martingale  stability  theorem,  the  following  characterization  of  the  limit  points  of  {pt(-)} 
can  be  established  as  in  [6,  Lemma  6.1.2]. 

Lemma  5.3.  Outside  a  set  of  zero  probability,  each  limit  point  fi  of  {pt{-)}  for  which  6^  >  0 
satisfies  fi*  =  rfv  . 

We  now  establish  the  existence  of  an  optimal  policy  under  (C3).  Since  the  proof  closely 
follows  the  steps  in  [6,  Theorem  6.1.1],  we  only  present  a  brief  sketch. 

Theorem  5.1.  Under  (C3),  there  exists  a  stable  Markov  policy  which  is  a.s.  optimal. 

Proof.  Let  vn  E  IIsat  be  such  that 

J  cdp[vn\  [  p*. 

We  extend  fi[vn]  to  V(M.d  x  S  x  U)  in  the  usual  manner  and  denote  it  also  by  fi[vn).  Let 
Poo  be  a  limit  point  of  {p[vn]}  and  denote  Voo  =  v ftoo,  where  v ^  is  obtained  from  p0 0  by 
the  decomposition  in  (5.6)  and  (5.8).  Then,  for  /  E  Co(Rd  x  S), 

E  /  LUn/(x’  ( dx ,  i)  =  y  [  Luf(x,  i)p[vn](dx ,  i,  du)  =  0  . 

its jRd  iesjR  xU 


Hence, 


Luf(x,  i)poo(dx,  i,  du)  =  0  . 
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Thus,  by  Lemmas  5.2  and  5.3,  =  rjVoo,  if  hIXrx  >  0.  Using  (C3),  we  can  demonstrate  as 

in  [6,  Lemma  6.1.3]  that  this  is  indeed  the  case.  Therefore, 


min 

«enSM 


J  cdp[v ]  =  J  cdp[v oo]  =  p* 


Finally,  following  the  technique  in  [6,  Lemma  6.1.3],  we  can  now  show  that  for  an  arbitrary 
policy  u, 

1  fT  , 

liminf  —  /  c(X(s),  S(s),  u(s))ds  >  p* ,  a.s. 

T—*oo  T  JQ 

which  establishes  the  optimality  of  Vqq  in  a  much  stronger  sense.  □ 

Theorem  5.2.  Under  (C3)  there  exists  a  v*  G  TIsmd  which  is  a.s.  optimal. 

Proof.  We  have  already  established  the  existence  of  G  Ll^jvf  which  is  a.s.  optimal.  We 
argue  as  in  [7,  p.  58].  Embed  I\  in  P(K  x  S  x  U)  by  assigning  zero  mass  at  {oo}  x  S  xU. 
Let  1 1  denote  the  closure  of  I\  in  "P(R  x  <S  x  U).  Then  Ii  is  a  compact  convex  set.  By 
Choquet’s  theorem  [23],  each  element  p  of  I\  is  the  barycenter  of  a  probability  measure 
m  supported  on  the  set  of  extreme  points  of  TV  Now,  each  extreme  point  of  Ii  must  be 
an  extreme  point  of  Jj,  since  otherwise  it  would  be  assigning  a  strictly  positive  mass  to 
{oo}  x  S  x  U.  If  m  assigns  a  strictly  positive  mass  to  extreme  points  of  I\,  which  are  not 
extreme  points  of  I\,  then  p  must  assign  a  strictly  positive  probability  to  {oo}  x  S  x  U, 
which  is  not  true.  Thus,  m  must  be  supported  on  the  set  If  consisting  of  the  extreme  points 
of  I\.  In  particular, 

di^j  m(dv) . 

It  follows  that  there  exists  a  v*  €  II smd  such  that 


J  cdp[v00]  =  J  cdp[v*} 

and  since  Uqo  G  U-sm  is  optimal,  the  optimality  of  v*  G  H-smd  follows.  □ 

We  now  investigate  the  existence  of  an  optimal  Markov  policy  under  the  blanket  stability 
conditions  in  (C1)-(C2). 

Lemma  5.4.  Under  either  (Cl)  or  (C2),  for  any  admissible  policy  v  G  n,  the  empirical 
process  {pt(v)}  defined  in  (5.5)  is  tight. 

The  proof  of  Lemma  5.4  closely  follows  the  arguments  in  the  proof  of  [6,  Theorem  6.2.2]. 
Topologize  the  space  II as  in  [6],  [14],  We  now  state  another  result,  the  proof  of  which 
closely  follows  [14,  Theorem  3.3,  Lemma  4.4]. 

Lemma  5.5.  Under  either  (Cl)  or  (C2),  the  sets  Ii,  Ii  are  compact  in  total  variation  and 
the  map  v  p[v\  (as  defined  in  (5.f))  is  continuous. 
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Theorem  5.3.  Under  either  (Cl)  or  (C2),  there  exists  a  v*  £  II smd  which  is  a.s.  optimal. 


Proof.  First  note  that  under  (Cl)  or  (C2),  Usm  =  and  II smd  ~  IIsd-  By  Lemma  5.5, 

there  exists  a  v  £  IIs m  such  that 


min 

sm 


J  cdp[v]  =  j"  cdp[v\. 


In  view  of  Lemma  5.4  and  the  decomposition  and  disintegration  of  the  measure  as  defined 
in  (5.6),  (5.8),  it  suffices  to  confine  our  attention  to  Usm  for  optimality.  Thus,  the  existence 
of  an  a.s.  optimal  v*  £  II smd  then  follows  via  Choquet’s  theorem  as  in  Theorem  5.2.  □ 


6.  Hamilton- Jacobi-Bellman  Equations.  In  this  section,  we  study  the  HJB  equations 
and  characterize  the  optimal  policy  in  terms  of  their  solution.  We  introduce  the  following 
condition: 


(C4)  The  cost  function  c  is  bounded,  continuous  and  Lipschitz  in  its  first  argument  uni¬ 
formly  with  respect  to  the  third. 


We  follow  the  vanishing  discount  approach,  i.e.,  we  derive  the  HJB  equations  for  the 
ergodic  criterion  by  taking  the  limit  of  the  HJB  equations  for  the  discounted  criterion  as 
the  discount  factor  approaches  zero.  The  results  and  the  broad  outline  of  these  proofs  follow 
those  of  [9].  However,  they  differ  in  important  technical  details. 

Let  Va(x,i )  denote  the  discounted  value  function  with  discount  factor  a  >  0,  i.e., 


(6.1)  Va(x,i)  =  inf  Evx  i 


—at 


U  o 


c(X(t),  S(t),u(t))dt 


x  e 


i  £  S . 


The  following  result  is  proved  in  [14]. 


Theorem  6.1.  Under  (Cf),  Va  is  the  unique  solution  in  G2(Rd  x  S)  P]Ct,(Rd  x  S )  of 
(6.2)  inf  {Lu Va(x,  i )  +  c(x,  i,u )}  =  aVa(x ,  i). 

u£U 


For  i  E  <S,  define 


(6.3) 


Gi  ■=  {x  €  Kd 

G  :=  1J  Gz . 


inf  c(x,i,u)  <  p*}, 

u£U 


i£S 

Observe  that  by  (C3),  G  is  compact. 

The  following  result  plays  a  very  crucial  role. 


Lemma  6.1.  Under  (CS)  and  (C4),  there  exists  ao  €  (0,1)  such  that  if  a  £  (0,  ao], 
inf  Va(x,i)  is  attained  on  the  set  G  as  defined  in  (6.3). 

(x,i)eRd*s 


Proof.  Let  va  £  Hmd  be  an  optimal  policy  for  the  discount  factor  a.  By  the  results  of  [14], 
for  i  £  S, 


(6.4)  '^r^bk(x,i,va{x,i))dVa}-X'  ^  ^  \jj(x,va(x,i))Va(x,  j)  +  c(x,  i,  va(x,  *)) 

k= l  °Xk  jes 

dVa(x,i) 

Jk\x,  t,  ut 

k=  1 


=  inf  l  bk(x,  i,  u)- 

u(EU  I  L J 


dxk 


+  ^  AaJ(.x,  u)  Va{x,  j)  +  c(x,  i,  u) 
jes 


a.e. 
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We  let  ||a:n||  — >  oo  in  Md  and  fix  i  G  S.  For  given  a,  let  ( Xn( •),  Sn( •))  be  the  process  under 
the  policy  va  with  Xn(0)  =  xn  and  S"(0)  =  i.  We  can  show  as  in  [20]  that  {Xn(-)  -  xn} 
are  tight  as  C([0,  oo);  Md)- valued  random  variables.  Dropping  to  a  subsequence  and  using 
Skorohod’s  theorem  [16.  p.  9]  we  may  assume  that  they  are  defined  on  a  common  probability 
space  and  converge  a.s.  in  C([0,  oo);  Mrf)  to  some  process  Y(-).  Hence,  ||Xn(f)||  ->  oo, 
uniformly  in  t  G  [0,T]  for  each  T  <  oo,  a.s.  By  (C3),  there  exist  e  >  0  and  M  >  0,  such 
that 

mf  {c(x,i,  u)}  >  p*  +  2e,  if  ||x||>M,  V  i  6  S  . 

We  select  a  constant  Ta  such  that 


(p*  +  2e)(l  -e  aTa)  >  p*  +s, 


i-e-»  e  "  <  WT2T)-  Since 
^  F'; 

it  follows  that 
(6.5) 


Va 

Xn 


rTa 


atc{Xn(t),Sn(t),va(Xn(t),Sn(t)))dt 


IJ  o 


p*  +£ 


a 


for  n  sufficiently  large.  On  the  other  hand,  by  a  standard  Tauberian  theorem, 


(6.6) 


limsup{aHQ(x,f)}  <  p* ,  V  (x,  i)  G  Rd  x  S . 

a— >0 


Fix  x0  €  Rd.  By  (6.6),  there  exists  a0  =  «o(^o)  such  that  Va(x0,-)  <  ^±1,  for  all 
a  <  a0.  Hence,  it  follows  from  (6.5)  that  if  a  <  a0,  then  inf  VJx.i)  is  attained  in  a  set 

x£Rd 

{x  e  Rd  :  ||x||  <  i?(a0)},  for  all  i  G  5.  Let 

xa<i  :=argmin {Va{x,i)}  ,  ( xa,ia )  :=  arg minify (xa)i,*)}  • 

xeMd  ies 

%  (6-2),  for  a  <  a0, 

(6J)  lc(xa>i,i,u)  +J2^i(x<*^u)V<*(x<*4i3)}  <  c*Va{xaA,i). 

^  jes  J 

and  from  (6.7), 


(6.8) 


^nf  ia7  n)  ^  fa) ,  V  oc  ^  cx o  . 


ueu 


We  claim  that  aVa(xa,ia)  <  p*  for  all  a  >  0.  Indeed,  for  any  v  G  USM, 


(6-9)  Va{x,i)<EZt 


■  poo 

/  e~atc(X(t ),  S(t ),  v(X(t),  S(t)))di 
Jo 


V  (a,  *)  G  Rd  x  5 . 
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Integrating  both  sides  of  (6.9)  with  respect  to  Tjv(dx,i)  and  using  Fubini’s  theorem,  we 
obtain 

E  /  Va{x,i)r)v{dx,i)  <  — . 

Hence, 


(6.10) 

From  (6.8), 


Ea(xa, 


inf  (c(xa5a,u 
uer/1  v 


)}  <P\ 


concluding  that  (xa,  ia)  G  G  x  5 .  □ 


Lemma  6.2.  Under  (C3)  and  (C4),  the  map  ( x,y,i,j )  \Va (x,i)  -  Va(y,j)\  is  bounded 

on  compact  subsets,  uniformly  in  a  €  (0,  a0]. 

Proof.  In  view  of  Lemma  6.1,  it  suffices  to  prove  that  Va(-,-)  =  Va(-,-)  -  Va{xa,ia )  is 
uniformly  bounded  on  compacta.  By  (6.2)  and  (6.4), 


LVaVa(x,i )  =  aVa{x,i)  -  c(x,i,va(x,i))  a.e. 


Let  R  >  0  be  large  enough  so  that  G  C  5(0,  R).  Let  (X(-),5(-))  be  the  process  under  the 
policy  va  and  define  r  =  inf {i  >  0  :  X(t)  <£  5(0,25)}.  Then  for  x  €  5(0,5),  using  the 
strong  Markov  property 


va(x,i)  =  5^ 


-  --  ^ 
Xtl 


LJo 

\r 

+  e: 


e-atc{X(t),S(t),va(X(t),S(t)))dt 

{c{X(t),S(t),va(X(t),  S(t )))  -  (X(r),  5(r))  }dt 

Va{X(T),S(r)) 


Thus, 


Va(x,i)-E^Va{X(r),S(r))\ 


=  EZ  jo  e~at{c{X(t),S(t),va(X(t),S(t)))  -  aVa(X(T),S(T))}dt  . 

Using  (C4)  and  Lemma  4.2,  we  deduce  that  there  exists  a  constant  C\  (independent  of  a) 
such  that 


(6.11)  \Va(x,i)-Evx-Va(X(r),S(T))\<C1,  V  (x,  i)  E  5(0,  5)  x  5  . 

We  write 

(6.12)  Va(x,i)-Va(xa,ia)  =  (va(x,i)  -5^Uq(X(t),5(t))) 

+  (5^14(X(r),5(r))  -  Va(xa,ia)) . 


23 


Let 


f(x,i)  =  Evx°Va(X(T),S(r))  -  Va(xa,ia). 

We  observe  that  /  >  0  and  LVa  f  =  0  in  W2'p[B(0,2R)  x  5),  2  <  p  <  oc.  Then,  by 
Theorem  4.1,  there  exists  a  constant  C2  (independent  of  a)  such  that,  in  view  of  (6.11), 

f(x,  i )  <  C2f(xa ,  ia)  <  CXC2  ,  V  (x,  i)  G  B{ 0,  R)  x  S . 

Hence, 

Hq(x,  «)  -  Va(xa,ia)  <  Ci(l  +  C2) ,  V  (x,i)  €  5(0,  R)  x  5.  □ 

Corollary  6.1.  For  any  e  >  0  and  any  compact  K  c  3Rd,  there  exists  ae  G  (0,  ao]  sttc/i 
that  for  all  x  £  K ,  i  £  S  and  a  G  (0,  ae), 

(6.13)  aVa(x,  i)  <  p*  +  e. 


Proof.  Follows  directly  from  Lemma  6.2  and  (6.10).  □ 

Theorem  6.2.  Under  (C3)  and  (Cf),  there  exists  a  function  V  G  C2(Rd  x  S)  and  a  scalar 
p  G  M  such  that  for  some  fixed  io  G  S, 

(6.14)  P<P*,  W (0,  zq)  =  0,  inf  V(x,i)  >  ~oo 

(x,i)€Rd  xS 

and  the  pair  (V,  p)  satisfies  the  HJB  equations  given  by 

(6.15)  inf  {LuV(x,i) +  c(x,i,u)}  =  p. 

Moreover,  among  all  pairs  (p,  p)  G  hF^(3Rd  x  5)  x  R,  2  <  p  <  oc,  satisfying  (6.15),  ( V ,  p*), 
is  the  unique  one  satisfying  (6.14). 

Proof.  Set  Va(x,i)  =  Va(x,i)  —  V^(0,io).  Then  V(0,  *o)  =  0  and  by  (6.2),  (6.4), 

LVaVa(x,i)  —  aVa(x,  i)  —  c(x,  i,va(x,  i)). 


By  Corollary  6.1,  Lemma  6.2  and  the  interior  estimates  for  solutions  of  uniformly  elliptic 
systems  [21,  pp.  398-402],  we  can  show  using  a  standard  bootstrap  argument  that  for  any 
R  >  0,  2  <  p  <  oo, 


sup 

ae(0,ctc) 


Va{;-) 


W2-p(B(0,R)xS ) 


<C, 


for  some  constant  C.  Since  is  compact  for  p  >  1,  {Va(-),a  G  (0,  ae)}  is 

sequentially  compact  in  .  Let  an  — >  0  in  (0.  aK).  By  dropping  to  a  subsequence, 

if  necessary,  let  Van  — >  V  in  for  some  V.  By  the  Sobolev  imbedding  theorem,  this 

convergence  is  also  uniform  on  compact  subsets  of  Rd.  Let  p  be  a  limit  point  of  anVan  (0,  i o) 
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and  hence  of  anVan(x,  i)  for  any  (x,t)  G  x  5,  in  view  of  Lemma  6.2.  By  (6.13),  p  <  p* . 
It  can  be  shown  as  in  [2],  [21,  p.  420]  that 


“jy  \  i^bk(x,i.u)- — ^  +  E  Xi3  (x,  u)Van  (x,  j)  +  c(x,  i,  u) 

k=l  k  j£S 


71—+00  U,(zU 


»£  I E  b*(x>  *»  +  E  m*.  u)v(*i  j) + c(x,  i,  u) 


k~l 


jes 


in  L?oc  strongly.  From  the  above  discussion,  it  follows  that  V  G  W/^,  for  any  2  <  p  <  oo, 
and  V  satisfies  (6.15)  in  D'  (i.e.,  in  the  sense  of  distributions).  By  elliptic  regularity, 
V  £  W 2  <  p  <  oo.  In  turn,  by  the  Sobolev  imbedding  theorem,  V  G  C1,7(Rd  x  5),  for 
0  <  7  <  1,  7  arbitrarily  close  to  1,  and  hence  by  (C4),  it  is  easy  to  see  that 


|  E  U^dXk  ^  +  E  Xi^x'  U)V(XJ)  +  c(x,  i,  u)  | 

|^fc=l  j£S  J 

is  in  C°’7(Rd  x  S).  By  elliptic  regularity  [15,  p.  287]  applied  to  (6.15),  we  conclude  that 
V  G  C2,7(Md  x  S).  Clearly,  V(0,i0)  =  0-  It  suffices  to  show  that  V  is  bounded  below.  For 
any  x  G  Md,  i  G  S , 


(6.16)  U(x,i)  =  lim  [Van(x,i)  —  Va  (0,io)] 

n — mx)  J 

-  nli^[^n(*an,f)  -  yQn(0,j0)]  +  lim o[Van{xan,ian)  -  Van{xan,i)]. 
Using  Lemmas  6.1  and  6.2,  it  follows  from  (6.16)  that  for  each  i  G  S, 


inf  V{x,  i)  >  — oo 

(x,i)^RdxS 

and  the  proof  of  the  first  part  of  the  theorem  is  complete.  The  second  assertion  can  be 
shown  by  following  the  methodology  in  [9],  □ 

Further,  based  on  Lemmas  6.1,  6.2  and  Theorem  4.1,  the  following  theorem  can  be  proved 
using  the  techniques  presented  in  [9].  We  therefore  skip  the  proof. 

Theorem  6.3.  Assume  (C3)  and  (C4).  Let  v*  G  UMd  be  such  that  for  each  i 

(6'17)  +E  Kix,u)V{x,j)  +  c(x,i,u) 


u£U 
d 


fc=  1 


jes 


=  E  (x,  i.  v *  (x,  *))  ^  2  K  (*,  v*(x,  *))  V  (x,  j )  +  c(x,  i,  v*  (x,  i))  a.  e. 


k= 1 


jes 


Then  v*  G  H-smd-  The  scalar  p  in  (6.15)  equals  p*  and  v*  is  a.s.  optimal.  Moreover, 
v  G  nsMjD  is  a.s.  optimal  if  and  only  if  it  satisfies  (6.17). 

Remark  6.1.  The  boundedness  condition  on  the  cost  function  c  may  be  relaxed.  For  un¬ 
bounded  c  we  can  use  a  suitable  truncation  procedure  to  approximate  c  by  a  sequence  of 
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bounded  functions.  Then  the  arguments  in  [9,  p.  202]  can  be  paralleled  to  establish  the 
results  in  Theorems  6. 2-6. 3. 

We  now  study  the  HJB  equation  under  (Cl)  and  (C4).  Recall  that  under  (Cl),  11m  = 
IIsm- 

Lemma  6.3.  Let  w  satisfy  (Cl).  Then  for  any  v  £  IT sm, 

(i)  /  w  (x,  i)rjv  (dx,  i)  <  oc. 
ies 

(ii)  lim^T[«;(X(t),5(f))]=0. 

Proof.  Let  R  >  0  and  tr  the  exit  time  of  X (t)  from  R(0,  R).  Then  by  Ito’s  formula 


TT'V 


w(X(t  A  tr),  S(t  A  tr))  —  w(x,i)  =  E'. 


7V 

Jx,i 


rt/\TR 


Lvw(X{s),S{s))ds 


Letting  R  — *  oo,  we  have 

E^lwiX^Sit))}  -w(x,i)  =  El  J  f  Lvw{X(s),S(s))ds 

Jo 


Therefore,  by  using  (Cl),  we  have 

~Ki Mm, $(*))]  <P- QKi Mm,s(t))] . 

Then  by  Gronwall’s  inequality, 

(6.18)  ££,[«,(*(*),  S(t))]  <Z+w(x,i)e-«. 

Both  (i)  and  (ii)  follow  directly  from  (6.18).  □ 

Lemma  6.4.  Assume  (Cl)  holds.  Let  a  >  0  be  such  that, 

Luw(x,  i)  <  —  1 ,  for  all  ||x||  >a,  u  £  U ,  i  £  S  . 


If 

(6.19)  ra  :=  inf{t  >  0  :  ||X(^)||  <  a}  , 
then,  for  all  v  £  II m,  ||z||  >  a  and  i  £  S, 

(6.20)  Evx  l  [ra]  <  w(x,  i ) . 


Proof.  Let  v  £  IIm-  Choose  R  >  0  such  that  a  <  ||rc||  <  R.  Let 

t'r  =  inf  {i  >  0  :  X(t)  £  B(0,  R)  \  B(0,  a)}. 
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Then  by  Ito’s  formula 


K 

Therefore, 

Thus, 

(6.21) 


rt/\TR 


15  0 


Lvw(X(s),  S(s))ds 


w(X{t  A  t'r),  S(t  A  =  w(x,i)  +  Evx  i 

w(X{tAT,R),S{tATR))  <  w{x,i)  -E^i[tAT'R]. 
Elz  [t  A  r*]  <  w(x,i). 


K.i 


Letting  first  t  —*  oo  and  then  R  — »  oo,  invoking  Fatou’s  lemma  at  each  step,  we  obtain 

(6.20).  □ 

Theorem  6.4.  Under  (Cl)  and  (C4),  the  HJB  equation  (6.15)  admits  a  unique  solution 
(V,  p)  in  the  class  x  5)f)  0{w),  satisfying  V (0,  i o)  =  0  for  some  fixed  io  6  S. 

Proof.  Let  v*  €  II smd  be  a.s.  optimal.  The  existence  of  such  a  u*  is  guaranteed  by 
Theorem  5.3.  Let 


K\  —  sup{c(x,  i,u)} 

x,i,w 

K[ 2  —  sup  /  cdp[v]. 
v£Usmd  J 

We  select  an  arbitrary  sequence  of  smooth  functions  t/>ra  :  Rd  — >  [0,  K\  +  452],  n  >  1,  that 
are  zero  on  5(0,  n)  and  equal  to  K\  +  ^K2  on  the  complement  of  5(0,  n  +  1),  and  define 


Cin(x,i,u)=  ±[c(x,i,u)  +  i>n(x)] 
c2n(x,  i,  u)  =  \  [ifn{x)  -  c(x,  i,  u)] . 


Then  for  a  sufficiently  large  n,  cin  and  c2n  both  satisfy  the  penalizing  condition  (C3).  We 
select  one  such  term  of  the  sequence  from  now  on  and  drop  the  subscript  n  for  notational 
convenience.  Let  (X(-),  £(•))  be  the  process  under  the  policy  v*.  For  a  >  0,  we  define 


Vn 


=  El 


,-at 


Cl(X(t),S(t),v*(X(t),S(t)))dt 


Va,2(a ;,i)  =  <i 


-at 


c2(X(t),S(t),v*(X(t),S(t)))dt 


Va(x,i)  =  El 


^  —  oct 


c(X(t),  S(t),  v*(X(t),  S(t)))dt 


Then  we  can  modify  the  arguments  in  the  proof  of  Lemma  6.2  to  conclude  that  for  a  fixed 
i o  6  S,  (Va7i(x,i)  —  14,i (0,  to))  and  (Vat2{x,i)  —  14,2(0,  io))  are  bounded  on  compacta 
uniformly  in  a  €  (0,  ckq],  for  some  an  >  0.  Hence, 


V4(^r,f)  ■  Va(x,i)  14(0)  io) 

=  [I4,i  (x,  i)  -  14,i(0,  i0)]  -  [14,2 (®,  i)  -14,2(0,  t0)] 
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is  bounded  on  compact  sets,  uniformly  in  a  E  (0,  ao]-  Arguing  as  in  the  proof  of  Theorem  6.2 
we  conclude  that  VQ(x,  i )  — »  V(x,i),  as  a  —*  0,  uniformly  on  compacta  and  in  W^( M.d  x <S), 
for  any  p  E  [2,  oo),  and  that  the  limit  V  satisfies 

Lv  V(x,i) +  c(x,i,v*(x,i))  =  p* , 

with  V(0,*o)  =  0.  Using  the  strong  Markov  property,  relative  to  the  stopping  time  ra  in 
(6.19),  we  obtain 

Va(x,i)  =  EZ'J  r  e-at[c(X(t),S(t),v*(X(t),S(t))]  -  aVQ(0,io))dt 
U  o 

+  Eii[e-^Va(X(ra),S(ra))  . 

Hence,  by  Lemma  6.4,  for  a  €  (0,  ao]  and  ||rc||  >  a, 

\Va(x,i)\<C1  +  C2EC[ra} 

<  C\  +  C2w(x ,  i), 


where  C i,  C2  are  positive  constants  independent  of  a.  Passing  to  the  limit  as  a  — >  0,  it 
follows  that  V  is  in  the  class  0(w).  Next  we  let  v  E  n,5M.D  be  such  that  for  each  i  E  S, 

h(x,  i,  v{x ,  t))  ^  +  ^2  Xij (x,  v(x,  i))V(x,  j)  +  c(x,  i,  v(x,  i)) 


d  ^  _  dV(x  i)  — 

=  inf  <> bk{x,i,u)  +  VA ij(x,u)V(x,j)  +  c(x,ipl 

ueu  z J  oxl-  * — * 


Suppose  that  for  some  i'  E  S,  there  exist  6  >  0,  such  that  the  set 


u  >  a.e. 


D  =  lx  E  Rd  :  V 


V ( x ,  i')  <  p*  —  c(x,  i  ,  v(x,  i'))  —  (5 1 


has  positive  Lebesgue  measure.  By  Ito’s  formula 


Eli[V[X{t),S(t))]-V{x,i)  =  Eli\ f  LvV(X(s),S(s))di 

.Jo 


This  is  justified  because  V  is  0(w).  Therefore, 


^[V{X(t),S(t))]  -V(x,i)  <  flp*  -c(X(s),S(s),v(X(s),S(s)))]ds 

Jo  J 

-6EvxJf  l{X(s)  e  D,S{s)  =  i'}ds  . 

Uo 


Dividing  by  t,  letting  t  — >  oo  and  using  Lemma  6.3,  we  have 


Pv  <  P*  ~  8r]v(D  X  {!'}). 
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By  Lemma  4.1,  r;„  ( D  x  {*'})  >  0.  Hence,  pv  <  p*  which  contradicts  the  optimality  of  v*. 
Thus,  for  each  i  E  S. 

(6.24)  inf  {LuV(x,  i)  +  c(x,  i,u)}  —  p*  a.e. 

Similar  arguments  as  in  the  proof  of  Theorem  6.2  establish  that  V  E  C2r<(WL  x  S ),  where 
0  <  7  <  1,  7  arbitrarily  close  to  1.  We  now  proceed  to  show  uniqueness.  Let  (V',p')  be 
another  solution  of  (6.15)  in  the  desired  class  satisfying  W/(0,  zo)  =  0.  Using  Ito’s  formula 
and  Lemma  6.3,  it  again  follows  that  p1  —  p* .  Therefore, 

Lv'(V\x,i)  -V{x,i))  >  0. 

Let  (X(t),S(t))  be  the  process  governed  by  v*  and  with  initial  law  rfo*.  Then, 

M(t)  :=  V'{X(t),S(t))-V{X(t),S(t)) 
is  a  submartingale  satisfying 

sup£T  \M(t)\  <  C[  +  C'2  [  w(x, i)r}v* (dx, i)  <  oo, 

t>°  ,6S  J  Rd 

by  Lemma  6.3,  where  C(,  are  suitable  constants.  Here  we  are  using  the  fact  that  both  V 
and  V'  are  of  O(w).  By  the  submartingale  convergence  theorem,  M (t)  converges  a.s.  Since 
(X(t),S(t))  is  ergodic  and  irreducible  under  v* ,  it  follows  that  V'(x,i)  —  V(x,i)  must  be 
constant  a.s.  This  constant  must  be  zero,  since  V'(0,  io)  —  V(0,io)  =  0.  □ 

Remark  6.2.  For  the  stable  case  we  have  carried  out  our  analysis  under  the  Lyapunov 
condition  (Cl).  Analogous  results  can  be  derived  under  the  condition  (C2). 

7.  Conclusions.  We  have  analyzed  the  optimal  control  of  switching  diffusions  with  a 
pathwise  average  cost  criterion.  Under  certain  conditions  we  have  established  the  existence 
of  a  stable,  nonrandomized  Markov  policy  which  is  a.s.  optimal  in  the  class  of  all  admis¬ 
sible  policies.  Also,  we  demonstrate  the  existence  of  a  unique  solution  to  the  associated 
HJB  equations  in  C2,  under  varying  conditions,  and  the  optimal  policy  is  characterized  as 
a  minimizing  selector  of  the  Hamiltonian.  We  have  applied  our  results  to  a  manufactur¬ 
ing  model  of  Bielecki  and  Kumar  and  have  exhibited  that  our  methodology  affords  both 
greater  generality  and  ease  of  solution.  By  studying  the  recurrence  and  ergodic  properties  of 
switching  diffusions  we  have  also  obtained  two  new  results  in  p.d.e.  viz.  a  strong  maximum 
principle  and  Harnack’s  inequality  for  a  weakly  coupled  elliptic  system. 

Acknowledgment.  The  authors  wish  to  thank  Prof.  S.R.S.  Varadhan  for  explaining  to  us 
the  work  of  Krylov  and  Safonov.  The  Appendix  in  this  paper  is  both  inspired  by  and  based 
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Appendix.  This  Appendix  is  devoted  to  the  proof  of  Theorem  4.1. 

Given  a  domain  17  c  Rd,  a  real  function  u  defined  on  17  x  S  is  viewed  as  a  vector  valued 
function  u  =  (u\, . . . ,  iijv),  with  each  component  Ui  being  a  real  function  on  17. 

Consider  a  second  order  operator  L  defined  by  (note  that  L/, -  is  different  from  the  operator 
in  (3.7)): 


(AT) 


(Lu)k(x)  :=  Lkuk(x )  +  ^2  Ckj(x)uj(x) ,  k  E  S 

jes 


A/c  . — 


d 

E' 

*>j= i 


vj 


{x) 


d 2 


dxjdx 


i=l 


E 

JG5 


Cjfej(x) 


Let  m,m,  7  and  be  given  positive  constants,  the  last  depending  on  the  choice  of  a 
bounded  domain  17.  We  denote  by  £  —  £(m,m,  7,  sp)  the  class  of  all  such  operators  L, 
with  coefficients  a •)•(•)  E  C0,1(Rd)  and  b'-(-),  ckj(-)  €  L°°(Rd),  satisfying: 
d 

(A.2)  m||C||2  <  ^  afj(x)(i(j  <  m||C||2,  for  all  s,(6  Md,  k  E  S  . 

i,j=  1 

(A. 3)  ||a^-(x) -aA(y)||oo  <  7||x-y||,  for  all  x,  y  G  Rd,  i,  j  E  { 1, . . .  ,  d],  k  E  S . 

(A.4)  Halloo  <  m,  ^2  llcfcjlloo  <  rn  and  ckj  >  0,  for  all  i  E  {l, . . .  ,  d},  j,  k  E  S  . 

j€S 

(A. 5)  The  matrix  C(x\£q)  :=  [ctj(x)  :  Cjj(x)  >  en\  is  irreducible  at  each  x  E  17. 

We  denote  by  iio  the  class  of  all  nonnegative  functions  u  E  W^(f?  x  S)  f)  C°(f2  x  5), 
satisfying  Lu  =  0  in  17,  for  some  L  E  £.  If  £  E  R,  then  it  >  £  is  to  be  interpreted  as  Ui  >  £, 
for  all  i  6  5,  and  if  £  =  (£1, . . .  ,£jv)  €  R^,  then  u  >  £  <=>  Ui  >  G,  for  all  i  G  5.  For 
better  clarity,  we  denote  all  M,v-valued  quantities  by  a  bold  letter.  Also  operations  such  as 
‘inf’  on  R^-valued  functions  are  meant  to  be  componentwise.  If  T  is  a  closed  subset  of  1? 
we  define,  for  x  G  17  and  £  G  K+ , 


:=  inf  {u(x)  :  u>^onf}. 

•ueiio 

Deviating  from  the  usual  vector  space  notation,  if  D  is  a  cube  in  Rd  and  8  >  0,  8D 
denotes  the  cube  which  is  concentric  to  D  and  whose  edges  are  8  times  as  long.  For  a 
measurable  set  A  c  Rd,  |T|  denotes  the  Lebesgue  measure  of  A,  while  13(A)  and  Ld(A) 
denote  the  sets  of  real-valued,  measurable  functions  on  A  such  that 


and 


||/IIb(A)  :=  esssup  |/(x)|  <  00 ,  V  /  G  B(A) 

x&A 


l- A  •■=  ^  \f(x)\ddx^j 


l/d 


<00,  V  /  G  Ld(A) 
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We  use  quite  frequently  the  following  comparison  principle,  which  can  be  viewed  as  a 
weaker  version  of  the  maximum  principle  in  that  it  holds  even  without  condition  (A.5  ):  If 
p,  £  W^(f2  x  S)  P)  C0(i7  x  S)  satisfy  Lp  <  Lip  in  1?  and  p  >  ip  on  dfi,  then  p  >  ip 
in  17.  The  same  comparison  principle  holds  for  p,  ip  £  W^(f2)  P|  C°(f2)  relative  to  the  set 
of  operators  {Lk}kes  as  defined  in  (A.l). 

We  start  with  a  measure  theoretic  result,  announced  in  [18]. 

Lemma  A.l.  Let  kcK^  be  a  cube,  F  C  K  a  closed  subset  and  0  <  a  <  1.  Define 
Q  :=  {Q  :  Q  is  a  subcube  of  K  and  |QP|r|  >  a\Q\] 

r-.=  U  (3£fW- 

QeQ. 

Then  either  f  =  K ,  or  |r|  >  ^|T|. 

Proof.  If  |T|  >  a\K\,  then  K  £  Q  and  jH  =  K.  So  we  assume  |T|  <  a\K\  or  equivalently, 
K  ^  Q.  We  subdivide  K  into  2d  congruent  subcubes  with  disjoint  interiors.  We  select  the 
ones  in  Q,  while  the  remaining  ones  are  similarly  subdivided  and  the  process  is  repeated 
indefinitely.  Let  Q0  be  the  collection  thus  obtained  and  with  Q  denoting  the  ancestor  of 
Q,  we  define 

?■■=  us. 

Qefio 

Clearly,  Q  C  3Q  f]  K\  hence,  f  Df.  Note  that,  discarding  repetitions,  F  can  be  represented 
as  a  disjoint  union  of  cubes  Q  which  are  not  in  Q.  Therefore,  each  member  Q  of  this  union 
satisfies  IQH-H  <  « | Q | :  and  by  cr-additivity  we  obtain 

\rnr\  <a\r\  <a\r\. 

By  the  regularity  properties  of  the  Lebesgue  measure  |Tf'|r,|  =  |T|  and  the  proof  is  com¬ 
plete.  □ 

Next  we  state  without  proof  a  variant  of  the  weak  maximum  principle  of  A.  D.  Aleksan- 
droff. 

Lemma  A. 2.  There  exist  constants  C\  >  0  and  ko  €  (0, 1]  such  that  if  D  C  Rd  is  any 
cube  of  volume  \D\  <  Ko  and  p  £  f]C°(D),  f  £  Ld(D )  satisfy  L^p  >  /  in  D  and 

p  =  0  on  dD,  for  some  L  £  £,  then 

sup{<p(:r)}  <  CilDI^H/lld;^  . 

x£D 


For  the  remainder  of  this  Appendix,  D  will  denote  an  open  cube  in  Md  of  volume  not 
exceeding  the  constant  k0  in  Lemma  A. 2. 
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Lemma  A. 3.  There  exist  constants  (3q  >  0  and  ao  <  1  such  that,  if  T  is  a  closed  subset 
of  some  cube  D  c  Kd,  satisfying  |_T|  >  ao|-D|,  then 

inf  ¥X(U  D,r-,t)>0oZ, 

x^D 

Proof.  Observe  that  if  u  €  ilp,  then  each  component  Uk  satisfies  Lkuk  <  0  in  D.  Define 
<f',  p"  E  Wf^(D)  D  C°(D)  by: 


Lkp'(x)  =  —Ir(x),  Lkp"{x)  =  -Ir<={x) ,  in  D 


and  p'(x)  =  p"{x)  =  0 ,  on  dD  . 

Then  p  \=  p1  +  p"  satisfies  Lkp  —  —  1  in  D  and  p  =  0  on  dD.  Without  loss  of  generality, 
suppose  D  is  centered  at  the  origin  and  consider  the  function 

d 

ip(x)  :=  n(\D\2/d  ~  ±x1). 

i=l 

Note  that  ip  —  0  on  dD  and  ip  >  0  in  D.  In  addition,  there  exists  a  positive  constant  C2 
such  that 

inf  iP(x)  >  C2\D\2/*\\LkiP\\B(D) ,  V  L  6  £ . 

xe±D 

Therefore,  by  the  comparison  principle, 

(A6) 

Using  Lemma  A. 2,  we  obtain 


p'  <  Ci\D\1/d\r\1/d  =  C1|D|2/ci(|^|)  d 

(A-7) 

p"  <  C\ \D\ l/<‘\rc\1/d  =  -  jg)  d. 

By  (A. 6)  and  (A. 7), 

p\x)  >  C2\D\2/d  -  C,\D\2'd{l  -  jg)  Vd,  V  x  6  \D . 

On  the  other  hand,  since  Lkp'  =  0  in  D\T  and  ip'  =  0  on  dD ,  the  comparison  principle 
yields 


(A. 8) 

Selecting  ao  to  satisfy 


inf  {uk(x)}>£k 
xe±D 


a0  >  1  - 


(A. 8)  yields 


inf  {ufe(x)j  > 


C2fk 

2Ci  ’ 


Hence,  the  claim  follows  with  fio  =  □ 
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Lemma  A. 4.  For  each  6  >  0,  there  exists  a  constant  k'g  >  0  such  that  if  Q  C  (1  —  S)D  is 
a  subcube  of  an  open  cube  D  C  M.d,  then 

9x{ilD,$Ql£)  >k'd,  V  x&3Qr\(l-S)D,  V  £  €  R?  . 


Proof.  Let  £?(r)  C  Md,  denote  the  ball  of  radius  r  centered  at  the  origin.  We  claim  that 
there  exists  a  constant  mo  >  0.  such  that  if  r  <  1,  then 


(A.9) 


inf  #x{iX B(r), £(;{);£)  >mo£,  V£e 


lN 


*eB(2p) 

In  order  to  establish  (A.9)  we  use  the  function 

ip(x)  :=  exp  | a  (l  -  -  1 ,  a:=^(16d  +  2),  x  €  B(r) , 

which  satisfies  Lk^p(x)  >  0  for  all  L  £  £,  provided  ||a:||  >  and  r  <  1.  By  the  comparison 
principle,  (A.9)  holds  with 

la  -i 
eie  —  1 

1 5a  7  • 

e  i6  —  1 

It  follows  that  if  B(r)  is  centered  at  y,  and  x  is  an  arbitrary  point  in  D  such  that  the 
distance  between  dD  and  the  line  segment  joining  x  and  y  is  at  least  r,  then 


(A.10) 


*x{HD,B{iy,Z)>(mo)et,  with  e  = 


4||x-y||-r 
2  r 


V  £  € 


p  TV 


Choosing  r  =  min{|,  |}|Q|1/d  and  applying  (A.  10),  an  easy  calculation  shows  that  the 
result  holds  with 


iJ  ._  *(*) 

KS  ■—  m0  I 


m  == 


6  \/d 


min{l,  £} 


□ 


Lemma  A. 5.  Suppose  that  there  exist  constants  £  and  6,  such  that  if  F  C  (1  —  8)D  is  a 
closed  subset  of  some  cube  D  and  £  £  R+,  then 


inf  (Up,  A;  £)  >  whenever  \F\>9\D\. 

xeiD 

Then  there  exists  a  constant  kg  >  0  such  that 

inf  T'x  T;  £)  >  ekg£,  whenever  \F\>ao6\D\, 

where  ao  is  the  constant  in  Lemma  A. 3. 

Proof.  Suppose  \T\  >  aa9\D\  and  let  y  €  T,  with  r  as  defined  in  Lemma  A.l  corresponding 
to  a  =  ao  and  K  =  (1  —  S)D.  Then  there  exists  a  subcube  Q  C  K  such  that  \F  fj  Q\  >  ao|Q| 
and  y  £  3 Q  fj  K.  We  use  the  identities, 

(A. ii)  *x(i xn,r-,z)>*JuD,r-  inf  *y{uD,r-z)) 

v  yer  ' 
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and 


(A. 12)  ¥y(UD,r',£)  >%(iXD)\Q]  inf  ¥z(ilDtr;£)) 

v  J 

>  Vy  (uD,  \Q-  vz (iig,  r n  Q;  £))  ■ 

From  Lemma  A. 3,  we  have 

(A. 13)  inf  <Pz(il0,rn<2;£)  >A)£- 

ze±Q 

From  Lemma  A. 4,  we  obtain  \Py  (ilp,  \Q]fio£)  >  ft ok'g£,  for  all  y  E  3Qf]K.  Hence, 
combining  (A.  12)  and  (A.  13)  yields 

(A.  14)  inf  <Fy  (ilD,  r;£)  >  ks£  ,  with  kg  (3ok's  . 

yer 

From  Lemma  A.l,  |T|  >  A-|F|  >  9\D\.  Therefore,  by  hypothesis, 

inf  \PX  (iXz>,  r ;  k6$)  >  ek6£  , 

xe±D 

which  along  with  (A.  11)  and  (A.  14)  yield  the  desired  result.  □ 

Theorem  A.l.  The  following  estimates  hold: 

(i)  Let  D  be  a  cube  and  T  C  (1  —  8)D  a  closed  subset.  Then  for  all  £  €  K+ , 

(A'15)  iinfD^(^.O0>«,({fl)'’<‘,«, 

where  the  constants  ao,  fio  and  ks  are  as  in  Lemma  A. 3  and  Lemma  A. 5. 

(ii)  There  exists  a  real  function  F  defined  in  [0, 1],  with  F(9)  >0,  if  9  >  0  such  that  if 
r  C  D  is  a  closed  subset  of  a  cube  D,  then 

(A. 16)  inf  tf*(.UD,r;£)  >F  (£[)$, 


Proof.  Part  (i)  is  direct  consequence  of  Lemmas  A. 3  and  A. 5.  For  part  (ii),  choose  6  = 
Then, 


(A.17) 

Since 


im(i  -m  >j£L -d-d- 6)*)  >  in  _  d*  >  m 

\d\  -  \d\  \  (1  b)  j  -\d\  -  m 


Vx(UD,r;£)  >¥x{iXD,rr[(l-6)D-,t), 

the  bound  in  (A. 16)  follows  from  (A. 15)  and  (A.17),  with 

F(9)  :=  (30  (f  Y{ld)  •  □ 

Definition  A.l.  If  A  C  L2  we  define  the  oscillation  of  a  function  u  E  C° ( Q  x  S)  over  A  by 
osc(«;  A)  =  max  sup{ufc(x)}  —  min  inf  {uk(x)\  . 

x^A.  x (L A 

The  oscillation  of  a  function  in  C°(  fL)  is  defined  in  the  usual  manner. 
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Theorem  A. 2.  If  D  is  a  cube,  u  G  51d  and  q  =  F{\),  with  F(-)  as  defined  in  Theorem  A.l 
(ii),  then 

osc (life;  ^D)  <  (l  —  |)  osc (it;  D) ,  V  fc  G  «S  . 


Proof.  Let 


sup  {ufc(x)}  ,  Ma  :=  max 

xe|£)  fcGS 

ml:—  inf  |«fc(x)},  m“  :=  min 
xe^D  kes 

and  Mb .  mb  be  the  corresponding  quantities  relative  to  D.  Consider  the  sets 

rfk)  :={ren  :  uk(x)  < 

r2(fc)  :={xeD  :  uk( x)  >  ^±^}. 

Suppose  |r2(fc)|  >  \\D\.  Since  u  —  mb  is  nonnegative  and  uk—mb  >  M  ~ 7—  in  T2fc' ,  applying 
Theorem  A.l  (ii),  yields 


uk{x)  —  mb  >  q - - - ,  Vre^D. 

Consequently,  mk  >  mb  +  qxl  j'Ti  and  since  Ma  <  Mb,  we  obtain 

(A. 18)  Ma -mak<Mb -mb  -  <  (l  -  §  ){Mb  -  mb) . 

On  the  other  hand,  if  \r^\  >  | |D|,  then  using  the  nonnegative  function  Mb  —  u.  we 
similarly  obtain 

(A. 19)  Mk  —  ma  <  (l  —  |)(Mb  —  mb) , 

and  the  result  follows  by  (A.18)-(A.19).  □ 

Theorem  A. 3.  There  exists  a  constant  M\  >  0  such  that,  for  any  u  G  il o, 
sup  {uAx) }  <  Mi  max  inf  |?ifc(x)},  V  i  £  5. 

lep  k€S 


Proof.  Let  fio  be  as  given  in  Lemma  A. 3  and  with  (){■)  and  q  as  in  (A.  15)  and  Theorem  A. 2, 
respectively,  define 


(A. 20) 


and 


qo  ■= 


(i-f) 

(i-!)' 


We  claim  that  the  value  of  the  constant  M i  may  be  chosen  as 


(A.21) 


M- 


i  •- 


Mo 
qfi  o 


27  N 


Vd 


2(7oP  -  1) 


Vp 
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We  argue  by  contradiction.  Suppose  u  6  ilp  violates  this  bound  and  let  . . .  , 

denote  the  points  in  ^ D  where  the  minima  of  u  are  attained,  i.e., 

inf  {'Ufc(x)}  =  Uk(x^) ,  keS. 


Without  loss  of  generality,  suppose  that  max{-Ufc(:r^)}  =  1  (u  can  always  be  scaled  to 

k£:S 

satisfy  this)  and  that  for  some  yo  6  and  ho  €  S,  uko(yo )  =  M  >  aM\,  with  a  >  1. 
Using  the  estimate  for  the  growth  of  the  oscillation  of  u  in  Theorem  A. 2,  we  will  show  that 
u  has  to  be  unbounded  in  ^D.  By  hypothesis,  ^  exceeds  M\  in  (A. 21)  and  in  order  to 
facilitate  the  construction  which  follows  we  choose  to  express  this  as 


(A. 22) 


1  i / j  /  4a  \Pv^/  1  \nP  1 

9  +  (?/3bAf)  S(~)  <  3  • 


For  £  >  0,  define 


:=  {ze\D  :  uk(z)  >  £}  ,  V :=  (J  Vk 


If  1  k  €  R+  stands  for  the  vector  whose  A;-th  component  is  equal  to  1  and  the  others  0,  then 


(A. 23) 


u(x^)  >  };£lfc)  ,  V  k  6  5, 


while,  on  the  other  hand,  Theorem  A.l  yields, 


(A. 24) 


pfiV(f) 


Cifc,  vk€S. 


By  (A.23)-(A,24)  and  using  (A. 20),  we  obtain  the  estimate 


(A. 25)  |2>«>|<£|24€)|<£; 


ufe(x(fc)) 


\D\  <  N 


\D\,  V£>0. 


Choosing  £  =  we  have  by  (A. 25) 


jx  G  \D  :  max{ufc(x)}  >  ^  j  <  N  ^ 


qpoM 


Hence,  if  Qo  is  a  cube  of  volume  |Qo|  =  qp\j)Pd\D\  centered  at  yo,  then 


(A. 26) 


osc(uko-,Q0)  >  (1  -  | )M . 


By  Theorem  A. 2,  we  obtain  from  (A. 26) 


(A. 27) 


(i- !) 


osc(u;3Q0)  >  n  _'UM  =  QoM . 

V>  2  / 
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Since  u  is  nonnegative,  (A. 27)  implies  that  there  exists  y (B  G  3Qo  and  k\  €  S  such  that 

ukl{yw)  >  q0M . 

Note  that  (A. 22)  implies  that  3Q0  C  \D.  Therefore,  we  can  repeat  the  argument,  now 
choosing  £  =  q0 ^  in  (A. 25)  and  a  cube  Q i  of  volume  N (qoqfi0~M)pd\D\  centered  at  y(l\  to 
conclude  that  there  exists  y ^  G  3Qi  and  ^  £  5  such  that  uk2{y ^)  >  q$M.  Inductively, 
we  can  construct  a  sequence  {y^n\  kn,  Qn}^=  0  satisfying,  for  all  n  =  0, 1, . . . , 

y{Q)  =yo&  l;Df)Qo,  y(n)  G  QnCl^Qn-l  , 

(A. 28)  IQn|1/d  =  iV1/d(i)nP(^)P|JD|1/fy 

ukn{y{n))  >q%M. 

The  inequality  in  (A. 22)  guarantees  that  G  for  all  n.  But  (A. 28)  implies  that  u  is 
unbounded  in  which  is  a  contradiction.  □ 

Remark  A.L  By  the  comparison  principle,  Lemmas  A.3-A.5  and  Theorem  A.l  clearly  hold 
unmodified  for  the  class  of  Lfc-superharmonic,  nonnegative  functions,  i.e.,  functions  u  G 
Wfoc(D)  Pi  C°(D),  satisfying  Lku  <  0  in  D,  for  some  k  G  S  and  L  G  £.  This  fact  will  be 
used  in  the  next  result. 

Lemma  A. 6.  Let  L  G  £,  k  G  S  and  suppose  ip  is  a  solution  to  the  Dirichlet  problem 
Lk'P  =  —  f  in  a  cube  D  c  Rd,  with  ip  —  0  on  dD,  with  f  satisfying 

0  <  f{x)  <  M ,  V  x  G  D  and  ||/||<i;D  >  £  >  0, 

for  some  constants  M  and  e.  Then  there  exists  a  constant  C'  =  C'(M,  e,  to,  to,  7)  such  that 

inf  {<p(x)}>C. 

165D 


Proof.  First  note  that  the  Dirichlet  problem  as  defined  has  a  unique  strong  solution  <p  G 
^loc  (D)  D  C°{D)i  for  all  p  G  [d,  00).  We  argue  by  contradiction.  Suppose  there  exists  a 
sequence  of  operators  {dJn>  C  £  and  a  sequence  of  functions  {/'n')  }^l1;  in  accord 
with  the  hypotheses  of  the  Lemma,  such  that  the  corresponding  solutions  of 

L^ip^  =  — satisfy 


inf  {<^(n)fy)}  < 


^3  D 


n* 


n  —  1,  2, . . . 


Thus,  by  Theorem  A.l, 


{xeD  :  ^nHx)>±} 


< 
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with  p  as  defined  in  (A. 20).  Since  the  sequence  ip^  is  bounded  in  L°°(D)  (by  Lemma  A. 2), 
it  follows  that  ip^  — »  0  in  LP(D),  as  n  — *  oo,  for  all  p  G  [1,  oo).  Let  D'  =  SD,  with  6  <  1, 
be  a  subcube  of  D ,  and  let  ||  •  | [2 ,P;D'  denote  the  standard  norm  of  W2'P{D').  We  use  the 
well  known  estimate 


■>{n)h,r,D'  <  C" (\\pn)\\viD  +  ||/(n)||KD)  , 


for  some  constant  C"  =  C" (\D\,p,6,d,  m,  m,  7) ,  to  conclude  that  the  first  and  second 
derivatives  of  ( p ^  converge  weakly  to  0  in  LP(D'),  for  all  p  G  [l,oo).  In  turn,  since 
Wq'p(D')  <—*  Wq’p(D')  is  compact  for  p  >  d,  using  the  standard  approximation  argument 
we  deduce  that  d^-x-  converges  in  LP(D')  strongly,  for  all  i  =  1  ,  ...,d.  Also,  since  the 
second-order  coefficients  of  are  uniformly  Lipschitz,  we  can  extract  a  subsequence, 
along  which  they  converge  uniformly.  Combining  all  the  previous  arguments,  we  deduce 
that  the  sequence  tp^}  converges  weakly  to  0  in  LP(D'),  p  G  [l,oo).  On  the  other 

/  \  l/d 

hand,  if  we  choose  <5  >  (  1  —  2m\d]  )  >  an  easy  calculation  yields, 

[  f{n\x)dx  >  n=  1,2, , 

J  D’  1 

resulting  in  a  contradiction.  □ 

We  pause  to  note  that  (A. 5)  has  not  been  utilized  in  any  of  the  results  obtained  thus 
far.  It  will  be  used  in  the  next  result  to  provide  the  necessary  ‘coupling’  between  distinct 
components  of  the  harmonic  function. 

Lemma  A. 7.  For  each  cube  D  C  ld  there  exists  a  constant  M2  >  0  such  that,  for  any 

u  G  it  D 

inf  {uAx)\  <  M2  inf  (u7-(x)}  ,  V  i,j  G  5. 


Proof.  Let  £d  be  the  constant  in  hypothesis  (A. 5).  Define  a  collection  of  functions 
{< pij{x ),  i,j  €  S}  C  W^(|D)  P|C°(|D),  relative  to  some  L  G  £,  by 


(A. 29) 


Lppij{x )  =  -Cij{x)  in  ~D  , 
tpij(x)  =  0,  if  i  =  j, 


and  Pij(x)  =  0  on  d(|l?)  , 


if  i^j 


and  let  &(x),  C(x )  denote  the  matrices  with  elements  { pij{x )}  and  {cij(x)} respec¬ 
tively.  By  (A. 4),  there  exists  a  constant  irreducible  matrix  Co  C  M.NxN,  with  elements 
equal  to  0  or  1  such  that 


(A. 30) 


{xG  \D  :  C(x)  >eDCD} 


> 


1 

N2  3d 


D 


It  follows,  by  (A. 29),  (A. 30)  and  Lemma  A. 6  that  there  exists  a  constant  e'D  >  0,  such  that 


(A.31) 


&{x)>e'DCD,  V  x  G  , 
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and  (A. 31)  holds  relative  to  any  Le£  used  to  generate  iptJ .  Therefore,  if  u  G  ilD  and  we 
define  u  inf  u(x)  and  u'  :=  inf  u(x).  it  is  a  direct  consequence  of  the  comparison 

principle  that 

(A. 32)  u(x )  >  &(x)u' ,  V  x  G  |-D  . 

On  the  other  hand,  by  Theorem  A.l, 

(A. 33)  yf  >F(^)u. 

By  (A.31)-(A.33), 

u(x)  >  e'DF(±r)CDu,  V  x  €  \D  , 

which  yields  u  >  ^'/jF  (^ijC])U.  In  turn,  the  irreducibility  of  Cd  implies  that 

ILi  >  (e'DF(^  Uj ,  Vi,jeS.  □ 

Combining  Theorem  A. 3  and  Lemma  A. 7  and  letting  M  :=  Mi  M2,  we  have: 

Theorem  A. 4.  For  each  cube  D  C  |D|  <  Kq  there  exists  a  constant  M  >  0  such  that, 
for  any  u  €  iXp 

Ui{y)  <  Muj(x) ,  V  x,y  £  ,  Vi,jeS. 

Theorem  4.1  easily  follows  from  Theorem  A. 4  by  covering  the  domain  f2  with  a  collection 
of  congruent  cubes  D  of  suitable  size.  For  an  elegant  exposition  of  this  technique  see  [11, 
pg.  153].  The  existence  of  a  constant  £q  >  0  satisfying  (A. 5)  is  guaranteed  by  the  continuity 
and  irreducibility  conditions  in  Assumption  3.1  (i)  and  (iii),  along  with  the  compactness  of 
U .  Concerning  (A. 3),  (A. 4)  and  the  upper  bound  in  (A. 2),  observe  that  for  each  bounded 
domain  17,  Assumption  3.1  (i)  implies  the  existence  of  constants  fh  and  7  satisfying  all  these 
conditions  in  17.  This  suffices  for  our  purposes. 
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